Original: HUMUS-Net: Hybrid Unrolled Multi-Scale Network Architecture for Accelerated MRI Reconstruction
Revised: HUMUS-Net: Hybrid Unrolled Multi-Scale Network Architecture for Accelerated MRI Reconstruction

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In accelerated MRI reconstruction, the anatomy of a patient is recovered from a setof under-sampled and noisy measurements.
Revised: In accelerated MRI reconstruction, the anatomy of a patient is recovered from a setof under-sampled and noisy measurements.

Original: Deep learning approaches have beenproven to be successful in solving this ill-posed inverse problem and are capable ofproducing very high quality reconstructions.
Revised: Deep learning approaches have beenproven to be successful in solving this ill-posed inverse problem and are capable ofproducing very high quality reconstructions.

Original: However, current architectures heavilyrely on convolutions, that are content-independent and have difﬁculties modelinglong-range dependencies in images.
Revised: However, current architectures heavilyrely on convolutions, that are content-independent and have difﬁculties modelinglong-range dependencies in images.

Original: Recently, Transformers, the workhorse ofcontemporary natural language processing, have emerged as powerful buildingblocks for a multitude of vision tasks.
Revised: Recently, Transformers, the workhorse ofcontemporary natural language processing, have emerged as powerful buildingblocks for a multitude of vision tasks.

Original: These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilizea self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures.
Revised: These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilizea self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures.

Original: However, Transformers incur extremelyhigh compute and memory cost when 1) the input image resolution is high and 2)when the image needs to be split into a large number of patches to preserve ﬁnedetail information, both of which are typical in low-level vision problems suchas MRI reconstruction, having a compounding effect.
Revised: However, Transformers incur extremelyhigh compute and memory cost when 1) the input image resolution is high and 2)when the image needs to be split into a large number of patches to preserve ﬁnedetail information, both of which are typical in low-level vision problems suchas MRI reconstruction, having a compounding effect.

Original: To tackle these challenges,we propose HUMUS-Net, a hybrid architecture that combines the beneﬁcial implicit bias and efﬁciency of convolutions with the power of Transformer blocksin an unrolled and multi-scale network.
Revised: To tackle these challenges,we propose HUMUS-Net, a hybrid architecture that combines the beneﬁcial implicit bias and efﬁciency of convolutions with the power of Transformer blocksin an unrolled and multi-scale network.

Original: HUMUS-Net extracts high-resolutionfeatures via convolutional blocks and reﬁnes low-resolution features via a novel
Revised: HUMUS-Net extracts high-resolutionfeatures via convolutional blocks and reﬁnes low-resolution features via a novel

Original: Transformer-based multi-scale feature extractor.
Revised: Transformer-based multi-scale feature extractor.

Original: Features from both levels are thensynthesized into a high-resolution output reconstruction.
Revised: Features from both levels are thensynthesized into a high-resolution output reconstruction.

Original: Our network establishesnew state of the art on the largest publicly available MRI dataset, the fastMRIdataset.
Revised: Our network establishesnew state of the art on the largest publicly available MRI dataset, the fastMRIdataset.

Original: We further demonstrate the performance of HUMUS-Net on two otherpopular MRI datasets and perform ﬁne-grained ablation studies to validate ourdesign.
Revised: We further demonstrate the performance of HUMUS-Net on two otherpopular MRI datasets and perform ﬁne-grained ablation studies to validate ourdesign.

Original:  Introduction
Revised:  Introduction

Original: Magnetic resonance imaging (MRI) is a medical imaging technique that uses strong magnetic ﬁeldsto picture the anatomy and physiological processes of the patient.
Revised: Magnetic resonance imaging (MRI) is a medical imaging technique that uses strong magnetic ﬁeldsto picture the anatomy and physiological processes of the patient.

Original: MRI is one of the most popularimaging modalities as it is non-invasive and doesn’t expose the patient to harmful ionizing radiation.
Revised: MRI is one of the most popularimaging modalities as it is non-invasive and doesn’t expose the patient to harmful ionizing radiation.

Original: The MRI scanner obtains measurements of the body in the spatial frequency domain also calledk-space .
Revised: The MRI scanner obtains measurements of the body in the spatial frequency domain also calledk-space .

Original: The data acquisition process is MR is often time-consuming.
Revised: The data acquisition process is MR is often time-consuming.

Original: Accelerated MRI
Revised: Accelerated MRI

Original: [Lustiget al., 2008] addresses this challenge by undersampling in the k-space domain, thus reducing thetime patients need to spend in the scanner.
Revised: [Lustiget al., 2008] addresses this challenge by undersampling in the k-space domain, thus reducing thetime patients need to spend in the scanner.

Original: However, recovering the underlying anatomy fromundersampled measurements is an ill-posed problem (less measurements than unknowns) and thusincorporating some form of prior knowledge is crucial in obtaining high quality reconstructions.
Revised: However, recovering the underlying anatomy fromundersampled measurements is an ill-posed problem (less measurements than unknowns) and thusincorporating some form of prior knowledge is crucial in obtaining high quality reconstructions.

Original: Classical MRI reconstruction algorithms rely on the assumption that the underlying signal is sparsein some transform domain and attempt to recover a signal that best satisﬁes this assumption in a technique known as compressed sensing (CS)
Revised: Classical MRI reconstruction algorithms rely on the assumption that the underlying signal is sparsein some transform domain and attempt to recover a signal that best satisﬁes this assumption in a technique known as compressed sensing (CS)

Original: [Candes et al., 2006, Donoho, 2006].
Revised: [Candes et al., 2006, Donoho, 2006].

Original: These classical
Revised: These classical

Original: CS techniques have slow reconstruction speed and typically enforce limited forms of image priors.
Revised: CS techniques have slow reconstruction speed and typically enforce limited forms of image priors.

Original: With the emergence of deep learning (DL), data-driven reconstruction algorithms have far surpassed
Revised: With the emergence of deep learning (DL), data-driven reconstruction algorithms have far surpassed

Original: CS techniques (see Ongie et al.
Revised: CS techniques (see Ongie et al.

Original: [2020] for an overview).
Revised: [2020] for an overview).

Original: DL models utilize large training datasetsto extract ﬂexible, nuanced priors directly from data resulting in excellent reconstruction quality.
Revised: DL models utilize large training datasetsto extract ﬂexible, nuanced priors directly from data resulting in excellent reconstruction quality.

Original: In recent years, there has been a ﬂurry of activity aimed at designing DL architectures tailoredto the MRI reconstruction problem.
Revised: In recent years, there has been a ﬂurry of activity aimed at designing DL architectures tailoredto the MRI reconstruction problem.

Original: The most popular models are convolutional neural networks(CNNs) that typically incorporate the physics of the MRI reconstruction problem, and utilize toolsfrom mainstream deep learning (residual learning, data augmentation, self-supervised learning).
Revised: The most popular models are convolutional neural networks(CNNs) that typically incorporate the physics of the MRI reconstruction problem, and utilize toolsfrom mainstream deep learning (residual learning, data augmentation, self-supervised learning).

Original: Comparing the performance of such models has been difﬁcult mainly due to two reasons.
Revised: Comparing the performance of such models has been difﬁcult mainly due to two reasons.

Original: First, therehas been a large variation in evaluation datasets spanning different scanners, anatomies, acquisitionmodels and undersampling patterns rendering direct comparison challenging.
Revised: First, therehas been a large variation in evaluation datasets spanning different scanners, anatomies, acquisitionmodels and undersampling patterns rendering direct comparison challenging.

Original: Second, medicalimaging datasets are often proprietary due to privacy concerns, hindering reproducibility.
Revised: Second, medicalimaging datasets are often proprietary due to privacy concerns, hindering reproducibility.

Original: More recently, the fastMRI dataset [Zbontar et al., 2019], the largest publicly available MRI dataset,has been gaining ground as a standard benchmark to evaluate MRI reconstruction methods.
Revised: More recently, the fastMRI dataset [Zbontar et al., 2019], the largest publicly available MRI dataset,has been gaining ground as a standard benchmark to evaluate MRI reconstruction methods.

Original: An annualcompetition, the fastMRI Challenge [Muckley et al., 2021], attracts signiﬁcant attention from themachine learning community and acts a driver of innovation in MRI reconstruction.
Revised: An annualcompetition, the fastMRI Challenge [Muckley et al., 2021], attracts signiﬁcant attention from themachine learning community and acts a driver of innovation in MRI reconstruction.

Original: However, over thepast years the public leaderboard has been dominated by a single architecture, the End-to-End VarNet
Revised: However, over thepast years the public leaderboard has been dominated by a single architecture, the End-to-End VarNet

Original: Sriram et al.
Revised: Sriram et al.

Original: [2020] 1 , with most models concentrating very closely around the same performancemetrics, hinting at the saturation of current architectural choices.
Revised: [2020] 1 , with most models concentrating very closely around the same performancemetrics, hinting at the saturation of current architectural choices.

Original: In this work, we propose HUMUS-Net: a Hybrid, UNrolled, Multi-Scale network architecturefor accelerated MRI reconstruction that combines the advantages of well-established architecturesin the ﬁeld with the power of contemporary Transformer-based models.
Revised: In this work, we propose HUMUS-Net: a Hybrid, UNrolled, Multi-Scale network architecturefor accelerated MRI reconstruction that combines the advantages of well-established architecturesin the ﬁeld with the power of contemporary Transformer-based models.

Original: We utilize the strongimplicit bias of convolutions, but also address their weaknesses, such as content-independence andinability to model long-range dependencies, by incorporating a novel multi-scale feature extractorthat operates over embedded image patches via self-attention.
Revised: We utilize the strongimplicit bias of convolutions, but also address their weaknesses, such as content-independence andinability to model long-range dependencies, by incorporating a novel multi-scale feature extractorthat operates over embedded image patches via self-attention.

Original: Moreover, we tackle the challenge ofhigh input resolution typical in MRI by performing the computationally most expensive operationson extracted low-resolution features.
Revised: Moreover, we tackle the challenge ofhigh input resolution typical in MRI by performing the computationally most expensive operationson extracted low-resolution features.

Original: HUMUS-Net establishes new state of the art in accelerated
Revised: HUMUS-Net establishes new state of the art in accelerated

Original: MRI reconstruction on the largest available MRI knee dataset.
Revised: MRI reconstruction on the largest available MRI knee dataset.

Original: At the time of writing this paper,
Revised: At the time of writing this paper,

Original: HUMUS-Net is the only Transformer-based architecture on the highly competitive fastMRI Public
Revised: HUMUS-Net is the only Transformer-based architecture on the highly competitive fastMRI Public

Original: Leaderboard.
Revised: Leaderboard.

Original: Our results are fully reproducible and the source code is available in the supplementary.
Revised: Our results are fully reproducible and the source code is available in the supplementary.

Original:  Background
Revised:  Background

Original: 2.1 Inverse Problem Formulation of Accelerated MRI Reconstruction
Revised: 2.1 Inverse Problem Formulation of Accelerated MRI Reconstruction

Original: An MR scanner obtains measurements of the patient anatomy in the frequency domain, also referredto as k-space .
Revised: An MR scanner obtains measurements of the patient anatomy in the frequency domain, also referredto as k-space .

Original: Data acquisition is performed via various receiver coils positioned around the anatomybeing imaged, each with different spatial sensitivity.
Revised: Data acquisition is performed via various receiver coils positioned around the anatomybeing imaged, each with different spatial sensitivity.

Original: Given a total number of N receiver coils, themeasurements obtained by the i th coil can be written as
Revised: Given a total number of N receiver coils, themeasurements obtained by the i th coil can be written as

Original: [Equation]
Revised: [Equation]

Original: where x ∗ ∈ C n is the underlying patient anatomy of interest, S i is a diagonal matrix that representsthe sensitivity map of the i th coil, F is a multi-dimensional Fourier-transform, and z i denotes themeasurement noise corrupting the observations obtained from coil i .
Revised: where x ∗ ∈ C n is the underlying patient anatomy of interest, S i is a diagonal matrix that representsthe sensitivity map of the i th coil, F is a multi-dimensional Fourier-transform, and z i denotes themeasurement noise corrupting the observations obtained from coil i .

Original: We use k = ( k 1 , ..., k N ) asa shorthand for the concatenation of individual coil measurements and x = ( x 1 , ..., x N ) as thecorresponding image domain representation after inverse Fourier transformation.
Revised: We use k = ( k 1 , ..., k N ) asa shorthand for the concatenation of individual coil measurements and x = ( x 1 , ..., x N ) as thecorresponding image domain representation after inverse Fourier transformation.

Original: Since MR data acquisition time is proportional to the portion of k-space being scanned, obtainingfully-sampled data is time-consuming.
Revised: Since MR data acquisition time is proportional to the portion of k-space being scanned, obtainingfully-sampled data is time-consuming.

Original: Therefore, in accelerated MRI scan times are reduced byundersampling in k-space domain.
Revised: Therefore, in accelerated MRI scan times are reduced byundersampling in k-space domain.

Original: The undersampled k-space measurements from coil i take theform
Revised: The undersampled k-space measurements from coil i take theform

Original: [Equation]
Revised: [Equation]

Original: where M is a diagonal matrix representing the binary undersampling mask, that has 0 values for allmissing frequency components that have not been sampled during accelerated
Revised: where M is a diagonal matrix representing the binary undersampling mask, that has 0 values for allmissing frequency components that have not been sampled during accelerated

Original: The forward model that maps the underlying anatomy to coil measurements can be written conciselyas ˜ k
Revised: The forward model that maps the underlying anatomy to coil measurements can be written conciselyas ˜ k

Original: = A ( x ∗ ) , where A ( · ) is the linear forward mapping and ˜ k is the stacked vector of allundersampled coil measurements.
Revised: = A ( x ∗ ) , where A ( · ) is the linear forward mapping and ˜ k is the stacked vector of allundersampled coil measurements.

Original: Our target is to reconstruct the ground truth object x ∗ fromthe noisy, undersampled measurements ˜ k .
Revised: Our target is to reconstruct the ground truth object x ∗ fromthe noisy, undersampled measurements ˜ k .

Original: Since we have fewer observations than variables torecover, perfect reconstruction in general is not possible.
Revised: Since we have fewer observations than variables torecover, perfect reconstruction in general is not possible.

Original: In order to make the problem solvable,prior knowledge on the underlying object is typically incorporated in the form of sparsity in sometransform domain.
Revised: In order to make the problem solvable,prior knowledge on the underlying object is typically incorporated in the form of sparsity in sometransform domain.

Original: This formulation, known as compressed sensing [Candes et al., 2006, Donoho,2006], provides a classical framework for accelerated MRI reconstruction [Lustig et al., 2008].
Revised: This formulation, known as compressed sensing [Candes et al., 2006, Donoho,2006], provides a classical framework for accelerated MRI reconstruction [Lustig et al., 2008].

Original: Inparticular, the above recovery problem can be formulated as a regularized inverse problem
Revised: Inparticular, the above recovery problem can be formulated as a regularized inverse problem

Original: [Equation]
Revised: [Equation]

Original: where R ( · ) is a regularizer that encapsulates prior knowledge on the object, such as sparsity in somewavelet domain.
Revised: where R ( · ) is a regularizer that encapsulates prior knowledge on the object, such as sparsity in somewavelet domain.

Original: 2.2 Deep Learning-based Accelerated MRI Reconstruction
Revised: 2.2 Deep Learning-based Accelerated MRI Reconstruction

Original: More recently, data-driven deep learning-based algorithms tailored to the accelerated MRI reconstruction problem have surpassed the classical compressed sensing baselines.
Revised: More recently, data-driven deep learning-based algorithms tailored to the accelerated MRI reconstruction problem have surpassed the classical compressed sensing baselines.

Original: Convolutional neuralnetworks trained on large datasets have established new state of the art in many medical imaging tasks.
Revised: Convolutional neuralnetworks trained on large datasets have established new state of the art in many medical imaging tasks.

Original: The highly popular U-Net [Ronneberger et al., 2015] and other similar encoder-decoder architectureshave proven to be successful in a range of medical image reconstruction [Hyun et al., 2018, Hanand Ye, 2018] and segmentation Çiçek et al.
Revised: The highly popular U-Net [Ronneberger et al., 2015] and other similar encoder-decoder architectureshave proven to be successful in a range of medical image reconstruction [Hyun et al., 2018, Hanand Ye, 2018] and segmentation Çiçek et al.

Original: [2016], Zhou et al.
Revised: [2016], Zhou et al.

Original: [2018] problems.
Revised: [2018] problems.

Original: In the encoderpath, the network learns to extract a set of deep, low-dimensional features from images via a series ofconvolutional and downsampling operations.
Revised: In the encoderpath, the network learns to extract a set of deep, low-dimensional features from images via a series ofconvolutional and downsampling operations.

Original: These concise feature representations are then graduallyupsampled and ﬁltered in the decoder to the original image dimensions.
Revised: These concise feature representations are then graduallyupsampled and ﬁltered in the decoder to the original image dimensions.

Original: Thus the network learns ahierarchical representation over the input image distribution.
Revised: Thus the network learns ahierarchical representation over the input image distribution.

Original: Unrolled networks constitute another line of work that has been inspired by popular optimizationalgorithms used to solve compressed sensing reconstruction problems.
Revised: Unrolled networks constitute another line of work that has been inspired by popular optimizationalgorithms used to solve compressed sensing reconstruction problems.

Original: These deep learning modelsconsist of a series of sub-networks, also known as cascades, where each sub-network correspondsto an unrolled iteration of popular algorithms such as gradient descent Zhang and Ghanem [2018]or ADMM Sun et al.
Revised: These deep learning modelsconsist of a series of sub-networks, also known as cascades, where each sub-network correspondsto an unrolled iteration of popular algorithms such as gradient descent Zhang and Ghanem [2018]or ADMM Sun et al.

Original: In the context of MRI reconstruction, one can view network unrollingas solving a sequence of smaller denoising problems, instead of the complete recovery problem inone step.
Revised: In the context of MRI reconstruction, one can view network unrollingas solving a sequence of smaller denoising problems, instead of the complete recovery problem inone step.

Original: Various convolutional neural networks have been employed in the unrolling frameworkachieving excellent performance in accelerated MRI reconstruction Putzky et al.
Revised: Various convolutional neural networks have been employed in the unrolling frameworkachieving excellent performance in accelerated MRI reconstruction Putzky et al.

Original: [2019], Hammerniket al.
Revised: [2019], Hammerniket al.

Original: E2E-VarNet [Sriram et al., 2020] is the current state-of-the-art convolutionalmodel on the fastMRI dataset.
Revised: E2E-VarNet [Sriram et al., 2020] is the current state-of-the-art convolutionalmodel on the fastMRI dataset.

Original: E2E-VarNet transforms the optimization problem in (2.1) to thek-space domain and unrolls the gradient descent iterations into T cascades, where the t th cascaderepresents the computation
Revised: E2E-VarNet transforms the optimization problem in (2.1) to thek-space domain and unrolls the gradient descent iterations into T cascades, where the t th cascaderepresents the computation

Original: [Equation]
Revised: [Equation]

Original: where ˆ k t is the estimated reconstruction in the k-space domain at cascade t , µ t is a learnable stepsize parameter and G ( · ) is a learned mapping representing the gradient of the regularization term in(2.1).
Revised: where ˆ k t is the estimated reconstruction in the k-space domain at cascade t , µ t is a learnable stepsize parameter and G ( · ) is a learned mapping representing the gradient of the regularization term in(2.1).

Original: The ﬁrst term is also known as data consistency (DC) term as it enforces the consistency of theestimate with the available measurements.
Revised: The ﬁrst term is also known as data consistency (DC) term as it enforces the consistency of theestimate with the available measurements.

Original:  Related Work
Revised:  Related Work

Original: Transformers in Vision – Vision Transformer (ViT) [Dosovitskiy et al., 2020], a fully nonconvolutional vision architecture, has demonstrated state-of-the-art performance on image classiﬁcation problems when pre-trained on large-scale image datasets.
Revised: Transformers in Vision – Vision Transformer (ViT) [Dosovitskiy et al., 2020], a fully nonconvolutional vision architecture, has demonstrated state-of-the-art performance on image classiﬁcation problems when pre-trained on large-scale image datasets.

Original: The key idea of ViT is to split theinput image into non-overlapping patches, embed each patch via a learned linear mapping and processthe resulting tokens via stacked self-attention and multi-layer perceptron (MLP) blocks.
Revised: The key idea of ViT is to split theinput image into non-overlapping patches, embed each patch via a learned linear mapping and processthe resulting tokens via stacked self-attention and multi-layer perceptron (MLP) blocks.

Revised: For moredetails we refer the reader to Appendix F and [Dosovitskiy et al., 2020].

Original: The beneﬁtof Transformers over convolutional architectures in vision lies in their ability to capture long-rangedependencies in images via the self-attention mechanism.
Revised: The beneﬁt of Transformersover convolutional architectures in vision lies in their ability to capture long-range dependencies inimages via the self-attention mechanism.

Original: Since the introduction of ViT, similar attention-based architectures have been proposed for many othervision tasks such as object detection [Carion et al., 2020], image segmentation [Wang et al., 2021b]and restoration [Cao et al., 2021b, Chen et al., 2021, Liang et al., 2021, Zamir et al., 2021, Wang et al.,
Revised: Since the introduction of ViT, similar attention-based architectures have been proposed for many othervision tasks such as object detection [Carion et al., 2020], image segmentation [Wang et al., 2021b] and restoration [Cao et al., 2021b, Chen et al., 2021, Liang et al., 2021, Zamir et al., 2021, Wang et al.

Original: 2021c].
Revised: ,2021c].

Original: A key challenge for Transformers in low-level vision problems is the quadratic computecomplexity of the global self-attention with respect to the input dimension.
Revised: A key challenge for Transformers in low-level vision problems is the quadratic computecomplexity of the global self-attention with respect to the input dimension.

Original: In some works, this issuehas been mitigated by splitting the input image into ﬁxed size patches and processing the patchesindependently [Chen et al., 2021].
Revised: In some works, this issuehas been mitigated by splitting the input image into ﬁxed size patches and processing the patchesindependently [Chen et al., 2021].

Original: Others focus on designing hierarchical Transformer architectures[Heo et al., 2021, Wang et al., 2021b] similar to popular ResNets
Revised: Others focus on designing hierarchical Transformer architectures[Heo et al., 2021, Wang et al., 2021b] similar to popular ResNets

Original: [He et al., 2015].
Revised: [He et al., 2015].

Original: Authors in Zamiret al.
Revised: Authors in Zamiret al.

Original: [2021] propose applying self-attention channel-wise rather than across the spatial dimensionthus reducing the compute overhead to linear complexity.
Revised: [2021] propose applying self-attention channel-wise rather than across the spatial dimensionthus reducing the compute overhead to linear complexity.

Original: Another successful architecture, the Swin
Revised: Another successful architecture, the Swin

Original: Transformer [Liu et al., 2021], tackles the quadratic scaling issue by computing self-attention onsmaller local windows.
Revised: Transformer [Liu et al., 2021], tackles the quadratic scaling issue by computing self-attention onsmaller local windows.

Original: To encourage cross-window interaction, windows in subsequent layers areshifted relative to each other.
Revised: To encourage cross-window interaction, windows in subsequent layers areshifted relative to each other.

Original: Transformers in Medical Imaging – Transformer architectures have been proposed recently to tacklevarious medical imaging problems.
Revised: Transformers in Medical Imaging – Transformer architectures have been proposed recently to tacklevarious medical imaging problems.

Original: Authors in Cao et al.
Revised: Authors in Cao et al.

Original: [2021a] design a U-Net-like architecturefor medical image segmentation where the traditional convolutional layers are replaced by Swin
Revised: [2021a] design a U-Net-like architecturefor medical image segmentation where the traditional convolutional layers are replaced by Swin

Original: Transformer blocks.
Revised: Transformer blocks.

Original: They report strong results on multi-organ and cardiac image segmentation.
Revised: They report strong results on multi-organ and cardiac image segmentation.

Original: In
Revised: In

Original: Zhou et al.
Revised: Zhou et al.

Original: [2021] a hybrid convolutional and Transformer-based U-Net architecture is proposedtailored to volumetric medical image segmentation with excellent results on benchmark datasets.
Revised: [2021] a hybrid convolutional and Transformer-based U-Net architecture is proposedtailored to volumetric medical image segmentation with excellent results on benchmark datasets.

Original: Similar encoder-decoder architectures for various medical segmentation tasks have been investigatedin other works [Huang et al., 2021, Wu et al., 2022].
Revised: Similar encoder-decoder architectures for various medical segmentation tasks have been investigatedin other works [Huang et al., 2021, Wu et al., 2022].

Original: However, these networks are tailored for imagesegmentation, a task less sensitive to ﬁne details in the input, and thus larger patch sizes are oftenused (for instance 4 in Cao et al.
Revised: However, these networks are tailored for imagesegmentation, a task less sensitive to ﬁne details in the input, and thus larger patch sizes are oftenused (for instance 4 in Cao et al.

Original: [2021a] ).
Revised: [2021a] ).

Original: This allows the network to process larger input images, asthe number of token embedding is greatly reduced, but as we demonstrate in Section 5.2 embeddingindividual pixels as 1 × 1 patches is crucial for MRI reconstruction.
Revised: This allows the network to process larger input images, asthe number of token embedding is greatly reduced, but as we demonstrate in Section 5.2 embeddingindividual pixels as 1 × 1 patches is crucial for MRI reconstruction.

Original: Thus, compute and memorybarriers stemming from large input resolutions are more severe in the MRI reconstruction task andtherefore novel approaches are needed.
Revised: Thus, compute and memorybarriers stemming from large input resolutions are more severe in the MRI reconstruction task andtherefore novel approaches are needed.

Original: Promising results have been reported employing Transformers in medical image denoising problems,such as low-dose CT denoising [Wang et al., 2021a, Luthra et al., 2021] and low-count PET/MRIdenoising
Revised: Promising results have been reported employing Transformers in medical image denoising problems,such as low-dose CT denoising [Wang et al., 2021a, Luthra et al., 2021] and low-count PET/MRIdenoising

Original: [Zhang et al., 2021].
Revised: [Zhang et al., 2021].

Original: However, these studies fail to address the challenge of poor scalingto large input resolutions, and only work on small images via either downsampling the originaldataset [Luthra et al., 2021] or by slicing the large input images into smaller patches [Wang et al.
Revised: However, these studies fail to address the challenge of poor scalingto large input resolutions, and only work on small images via either downsampling the originaldataset [Luthra et al., 2021] or by slicing the large input images into smaller patches [Wang et al.

Original: ,2021a].
Revised: ,2021a].

Original: In contrast, our proposed architecture works directly on the large resolution images that oftenarise in MRI reconstruction.
Revised: In contrast, our proposed architecture works directly on the large resolution images that often arise in MRI reconstruction.

Original: Even though some work exists on Transformer-based architectures for supervised accelerated MRI reconstruction [Huang et al., 2022, Lin and Heckel, 2022, Feng et al.
Revised: Even though some work exists on Transformer-based architectures forsupervised accelerated MRI reconstruction [Huang et al., 2022, Lin and Heckel, 2022, Feng et al.

Original: ,2021],  to the best of our knowledge ours is the ﬁrst work to demonstrate state-of-the-art results onlarge-scale MRI datasets such as the fastMRI dataset.
Revised: ,2021], and for unsupervised pre-trained reconstruction [Korkmaz et al., 2022], to the best of ourknowledge ours is the ﬁrst work to demonstrate state-of-the-art results on large-scale MRI datasetssuch as the fastMRI dataset.

Original:  Method
Revised:  Method

Original: HUMUS-Net combines the efﬁciency and beneﬁcial implicit bias of convolutional networks with thepowerful general representations of Transformers and their capability to capture long-range pixeldependencies.
Revised: HUMUS-Net combines the efﬁciency and beneﬁcial implicit bias of convolutional networks with thepowerful general representations of Transformers and their capability to capture long-range pixeldependencies.

Original: The resulting hybrid network processes information both in image representation (viaconvolutions) and in patch-embedded token representation (via Transformer blocks).
Revised: The resulting hybrid network processes information both in image representation (viaconvolutions) and in patch-embedded token representation (via Transformer blocks).

Original: Our proposedarchitecture consists of a sequence of sub-networks, also called cascades, each of which represents anunrolled iteration of an underlying optimization algorithm.
Revised: Our proposedarchitecture consists of a sequence of sub-networks, also called cascades. Each cascade representsan unrolled iteration of an underlying optimization algorithm in k-space (see (2.2)), with an imagedomain denoiser, the HUMUS-Block.

Original: First, we describe the architecture of the HUMUS-Block, the core component in the sub-network.
Revised: First, we describe the architecture of the HUMUS-Block, thecore component in the sub-network.

Original: Then, we specify the high-level, unrolledarchitecture of HUMUS-Net in Section 4.3.
Revised: Then, we specify the high-level, k-space unrolling architectureof HUMUS-Net in Section 4.3.

Original: 4.1 HUMUS-Block Architecture
Revised: 4.1 HUMUS-Block Architecture

Original: The HUMUS-Block acts as an image-space denoiser that receives an intermediate reconstruction fromthe previous cascade and performs a single step of denoising to produce an improved reconstructionfor the next cascade.
Revised: The HUMUS-Block acts as an image-space denoiser that receives an intermediate reconstruction fromthe previous cascade and performs a single step of denoising to produce an improved reconstructionfor the next cascade.

Original: It extracts high-resolution, shallow features and low-resolution, deep featuresthrough a novel multi-scale transformer-based block, and synthesizes high-resolution features fromthose.
Revised: It extracts high-resolution, shallow features and low-resolution, deep featuresthrough a novel multi-scale transformer-based block, and synthesizes high-resolution features fromthose.

Original: The high-level overview of the HUMUS-Block is depicted in Fig.
Revised: The high-level overview of the HUMUS-Block is depicted in Fig.

Original: High-resolution Feature Extraction– The input to our network is a noisy complex-valued imagex in ∈ R h × w × c in derived from undersampled k-space data, where the real and imaginary partsof the image are concatenated along the channel dimension.
Revised: High-resolution Feature Extraction– The input to our network is a noisy complex-valued imagex in ∈ R h × w × c in derived from undersampled k-space data, where the real and imaginary partsof the image are concatenated along the channel dimension.

Original: First, we extract high-resolutionfeatures F H ∈ R h × w × d H from the input noisy image through a convolution layer f H written as
Revised: First, we extract high-resolutionfeatures F H ∈ R h × w × d H from the input noisy image through a convolution layer f H written as

Original: F H = f H ( x in ) .
Revised: F H = f H ( x in ) .

Original: This initial 3 × 3 convolution layer provides early visual processing at a relativelylow cost and maps the input to a higher, d H dimensional feature space.
Revised: This initial 3 × 3 convolution layer provides early visual processing at a relativelylow cost and maps the input to a higher, d H dimensional feature space.

Original: Important to note that theresolution of the extracted features is the same as the spatial resolution of the input image.
Revised: Important to note that theresolution of the extracted features is the same as the spatial resolution of the input image.

Original: Low-resolution Feature Extraction– In case of MRI reconstruction, this resolution is typicallysigniﬁcantly higher than commonly used image datasets ( 32 × 32 - 256 × 256 ), posing  signiﬁcantchallenge to contemporary Transformer-based models.
Revised: Low-resolution Feature Extraction– In case of MRI reconstruction, this resolution is typicallysigniﬁcantly higher than commonly used image datasets ( 32 × 32 - 256 × 256 ), posing a signiﬁcantchallenge to contemporary Transformer-based models.

Original: Thus we apply a convolutional featureextractor f L to obtain low-resolution features F L = f L ( F H ) , with F L ∈ R
Revised: Thus we apply a convolutional featureextractor f L to obtain low-resolution features F L = f L ( F H ) , with F L ∈ R

Original: h L × w L × d L where f Lconsists of a sequence of convolutional blocks and spatial downsampling operations.
Revised: h L × w L × d L where f Lconsists of a sequence of convolutional blocks and spatial downsampling operations.

Original: The speciﬁcarchitecture is depicted in Figure 4.
Revised: The speciﬁcarchitecture is depicted in Figure 4.

Original: The purpose of this module is to perform deeper visual processingand to provide manageable input size to the subsequent computation and memory heavy hybridprocessing module.
Revised: The purpose of this module is to perform deeper visual processingand to provide manageable input size to the subsequent computation and memory heavy hybridprocessing module.

Original: In this work we choose h L = h 2 and w L = w 2 which strikes a balance betweenpreserving spatial information and resource demands.
Revised: In this work, we choose h L = h 2 and w L = w 2 which strikes a balance betweenpreserving spatial information and resource demands.

Original: Furthermore, in order to compensate for thereduced resolution we increase the feature dimension to d L = 2 · d H :=
Revised: Furthermore, in order to compensate for thereduced resolution we increase the feature dimension to d L = 2 · d H :=

Original: d .
Revised: d .

Original: Deep Feature Extraction– The most important part of our model is MUST, a MU lti-scale residual
Revised: Deep Feature Extraction– The most important part of our model is MUST, a MU lti-scale residual

Original: S win T ransformer network.
Revised: S win T ransformer network.

Original: MUST is a multi-scale hybrid feature extractor that takes the lowresolution image representations F L and performs hierarchical Transformer-convolutional hybridprocessing in an encoder-decoder fashion, producing deep features F D = f D ( F L ) , where the speciﬁcarchitecture behind f D is detailed in Subsection 4.2.
Revised: MUST is a multi-scale hybrid feature extractor that takes the lowresolution image representations F L and performs hierarchical Transformer-convolutional hybridprocessing in an encoder-decoder fashion, producing deep features F D = f D ( F L ) , where the speciﬁcarchitecture behind f D is detailed in Subsection 4.2.

Original: High-resolution Image Reconstruction– Finally, we combine information from shallow, highresolution features F H and deep, low-resolution features F D to reconstruct the high-resolutionresidual image via a convolutional reconstruction module f R .
Revised: High-resolution Image Reconstruction– Finally, we combine information from shallow, highresolution features F H and deep, low-resolution features F D to reconstruct the high-resolutionresidual image via a convolutional reconstruction module f R .

Original: The residual learning paradigmallows us to learn the difference between noisy and clean images and helps information ﬂow withinthe network [He et al., 2015].
Revised: The residual learning paradigmallows us to learn the difference between noisy and clean images and helps information ﬂow withinthe network [He et al., 2015].

Original: Thus the ﬁnal denoised image x out ∈ R h × w × c in is obtained asx out = x in + f R ( F H , F D ) .
Revised: Thus the ﬁnal denoised image x out ∈ R h × w × c in is obtained asx out = x in + f R ( F H , F D ) .

Original: The speciﬁc architecture of the reconstruction network is depicted in
Revised: The speciﬁc architecture of the reconstruction network is depicted in

Original: Figure 4.
Revised: Figure 4.

Original: 4.2 Multi-scale Hybrid Feature Extraction viaMUST
Revised: 2 Multi-scale Hybrid Feature Extraction via

Original: The key component to our architecture is MUST, a multi-scale hybrid encoder-decoder architecturethat performs deep feature extraction in both image and token representation (Figure 1, bottom).
Revised: The key component to our architecture is MUST, a multi-scale hybrid encoder-decoder architecturethat performs deep feature extraction in both image and token representation (Figure 1, bottom).

Original: First, individual pixels of the input representation of shape h 2 × w 2 × d are transformed into h 2 · w 2 tokens of dimension d via learned linear mapping and positional encoding.
Revised: First,individual pixels of the input representation of shape h 2 × w 2 × d are ﬂattened and passed through alearned linear mapping to yield h 2 · w 2 tokens of dimension d .

Original: Tokens corresponding todifferent image patches are subsequently merged in the encoder path, resulting in a concise latentrepresentation.
Revised: Tokens corresponding to different imagepatches are subsequently merged in the encoder path, resulting in a concise latent representation.

Original: This highly descriptive representation is passed through a bottleneck block andprogressively expanded by combining tokens from the encoder path via skip connections.
Revised: Thishighly descriptive representation is passed through a bottleneck block and progressively expanded bycombining tokens from the encoder path via skip connections.

Original: The ﬁnaloutput is rearranged to match the exact shape of the input low-resolution features F L , yielding a deepfeature representation F D .
Revised: The ﬁnal output is rearranged to matchthe exact shape of the input low-resolution features F L , yielding a deep feature representation F D .

Original: Our design is inspired by the success of Residual Swin Transformer Blocks (RSTB) in imagedenoising and super-resolution
Revised: Our design is inspired by the success of Residual Swin Transformer Blocks (RSTB) in imagedenoising and super-resolution

Original: [Liang et al., 2021].
Revised: [Liang et al., 2021].

Original: RSTB features a stack of Swin Transformerlayers (STL) that operate on tokens via a windowed self-attention mechanism [Liu et al., 2021],followed by convolution in image representation.
Revised: RSTB features a stack of Swin Transformerlayers (STL) that operate on tokens via a windowed self-attention mechanism [Liu et al., 2021],followed by convolution in image representation.

Original: However, RSTB blocks operate on a single scale,therefore they cannot be readily applied in a hierarchical encoder-decoder architecture.
Revised: However, RSTB blocks operate on a single scale,therefore they cannot be readily applied in a hierarchical encoder-decoder architecture.

Original: Therefore,we design three variations of RSTB to facilitate multi-scale processing as depicted in Figure 2.
Revised: Therefore,we design three variations of RSTB to facilitate multi-scale processing as depicted in Figure 2.

Original: RSTB-B is the bottleneck block responsible for processing the encoded latent representation whilemaintaining feature dimensions.
Revised: RSTB-B is the bottleneck block responsible for processing the encoded latent representation whilemaintaining feature dimensions.

Original: Thus, we keep the default RSTB architecture for our bottleneckblock, which already operates on a single scale.
Revised: Thus, we keep the default RSTB architecture for our bottleneckblock, which already operates on a single scale.

Original: RSTB-D has a similar function to convolutional downsampling blocks in U-Nets, but it operateson embedded tokens.
Revised: RSTB-D has a similar function to convolutional downsampling blocks in U-Nets, but it operates onembedded tokens.

Original: Given an input with size h i · w i × d , we pass it through an RSTB-B blockand apply PatchMerge operation.
Revised: Given an input with size h i · w i × d , we pass it through an RSTB-B block and apply PatchMerge operation.

Original: PatchMerge linearly combines tokens corresponding to 2 × 2non-overlapping image patches, while simultaneously increasing the embedding dimension (see 
Revised: PatchMerge linearly combines tokens corresponding to 2 × 2 non-overlappingimage patches, while simultaneously increasing the embedding dimension (see Figure 3, top and

Original: Figure 3, top) resulting in an output of size h i 2 · w i 2 × 2 · d .
Revised: Appendix H for more details) resulting in an output of size h i 2 · w i 2 × 2 · d .

Original: Furthermore, RSTB-D outputs the higherdimensional representation before patch merging to be subsequently used in the decoder path viaskip connection.
Revised: Furthermore, RSTB-Doutputs the higher dimensional representation before patch merging to be subsequently used in thedecoder path via skip connection.

Original: RSTB-U used in the decoder path is analogous to convolutional upsampling blocks.
Revised: RSTB-U used in the decoder path is analogous to convolutional upsampling blocks.

Original: An input withsize h
Revised: An input withsize h

Original: i · w i × d is ﬁrst expanded into a larger number of lower dimensional tokens through a linearmapping via PatchExpand (see Figure 3, bottom).
Revised: i · w i × d is ﬁrst expanded into a larger number of lower dimensional tokens through a linearmapping via PatchExpand (see Figure 3, bottom and Appendix H for more details).

Original: PatchExpand reverses the effect of PatchMerge onfeature size, thus resulting in 2 h i · 2 w i tokens of dimension d 2 .
Revised: PatchExpandreverses the effect of PatchMerge on feature size, thus resulting in 2 h i · 2 w i tokens of dimension d 2 .

Original: Next, we mix information from theobtained expanded tokens with skip embeddings from higher scales via TokenMix .
Revised: Next, we mix information from the obtained expanded tokens with skip embeddings from higherscales via TokenMix .

Original: This operationlinearly combines tokens from both paths and normalizes the resulting vectors.
Revised: This operation linearly combines tokens from both paths and normalizes theresulting vectors.

Original: Finally, the mixedtokens are processed by an RSTB-B block.
Revised: Finally, the mixed tokens are processed by an RSTB-B block.

Original: 4.3 Iterative Unrolling
Revised: 4.3 Iterative Unrolling

Original: Architectures derived from unrolling the iterations of various optimization algorithms have proven tobe successful in tackling various inverse problems including MRI reconstruction.
Revised: Architectures derived from unrolling the iterations of various optimization algorithms have proven tobe successful in tackling various inverse problems including MRI reconstruction.

Original: These architecturecan be interpreted as a cascade of simpler denoisers, each of which progressively reﬁnes the estimatefrom the preceding unrolled iteration.
Revised: These architecturecan be interpreted as a cascade of simpler denoisers, each of which progressively reﬁnes the estimatefrom the preceding unrolled iteration.

Original: Following Sriram et al.
Revised: Following Sriram et al.

Original: [2020], we unroll the gradient descent iterations of the inverse problem in (2.1)in k-space domain, yielding the iterative update scheme in (2.2).
Revised: [2020], we unroll the gradient descent iterations of the inverse problem in (2.1)in k-space domain, yielding the iterative update scheme in (2.2).

Original: We apply regularization in imagedomain via our proposed HUMUS-Block, that is we have G ( k ) = F (cid:0) E (cid:0) D (cid:0) R (cid:0) F − 1 ( k ) (cid:1)(cid:1)(cid:1)(cid:1) , where
Revised: We apply regularization in image

Original: D denotes the HUMUS-Block, R ( x 1 , ..., x N ) =

Original: (cid:80) Ni

Original: =1 S ∗

Original: i x

Original: i is the reduce operator that combinescoil images via the corresponding sensitivity maps and E ( x ) =

Original: ( S 1 x , ..., S N x ) is the expandoperator that maps the combined image back to individual coil images.

Original: The sensitivity maps can be

Original: 4.4 Adjacent Slice Reconstruction (ASR)
Revised: 4.4 Adjacent Slice Reconstruction (ASR)

Original: We observe improvements in reconstruction quality when instead of processing the undersampleddata slice-by-slice, we jointly reconstruct a set of adjacent slices via HUMUS-Net
Revised: We observe improvements in reconstruction quality when instead of processing the undersampleddata slice-by-slice, we jointly reconstruct a set of adjacent slices via HUMUS-Net

Original: (Figure 5).
Revised: (Figure 5).

Original: That is,if we have a volume of undersampled data ˜ k vol =
Revised: That is,if we have a volume of undersampled data ˜ k vol =

Original: (˜ k 1 , ..., ˜ k K ) with K slices, when reconstructingslice c , we instead reconstruct the volume (˜ k c − a , ..., ˜ k
Revised: (˜ k 1 , ..., ˜ k K ) with K slices, when reconstructingslice c , we instead reconstruct the volume (˜ k c − a , ..., ˜ k

Original: c − 1 , ˜ k c , ˜ k
Revised: c − 1 , ˜ k c , ˜ k

Original: c +1 , ..., ˜ k
Revised: c +1 , ..., ˜ k

Original: c + a ) by concatenating themalong the coil channel dimension, where a denotes the number of adjacent slices added on each side.
Revised: c + a ) by concatenating themalong the coil channel dimension, where a denotes the number of adjacent slices added on each side.

Original: However, we only calculate and backpropagate the loss on the center slice c of the reconstructedvolume.
Revised: However, we only calculate and backpropagate the loss on the center slice c of the reconstructedvolume.

Original: The beneﬁt of ASR is that the network can remove artifacts corrupting individual slices as itsees a larger context of the slice by observing its neighbors.
Revised: The beneﬁt of ASR is that the network can remove artifacts corrupting individual slices as itsees a larger context of the slice by observing its neighbors.

Original: Even though ASR increases computecost, it is important to note that it does not impact the number of token embeddings (spatial resolutionis unchanged) and thus can be combined favorably with Transformer-based methods.
Revised: Even though ASR increases computecost, it is important to note that it does not impact the number of token embeddings (spatial resolutionis unchanged) and thus can be combined favorably with Transformer-based methods.

Original:  Experiments
Revised:  Experiments

Original: In this section we provide experimental results on our proposed architecture, HUMUS-Net.
Revised: In this section we provide experimental results on our proposed architecture, HUMUS-Net.

Original: First,we demonstrate the reconstruction performance of our model on various datasets, including thelarge-scale fastMRI dataset.
Revised: First,we demonstrate the reconstruction performance of our model on various datasets, including thelarge-scale fastMRI dataset.

Original: Then, we justify our design choices through a set of ablation studies.
Revised: Then, we justify our design choices through a set of ablation studies.

Original: Our source code is available in the supplementary.
Revised: Our source code is available in the supplementary.

Original: 5.1 Benchmark Experiments
Revised: 5.1 Benchmark Experiments

Original: We investigate the performance of HUMUS-Net on three different datasets.
Revised: We investigate the performance of HUMUS-Net on three different datasets.

Original: We use the structuralsimilarity index metric (SSIM)  as a basis of our evaluation, which is the most common evaluationmetric in medical image reconstruction.
Revised: We use the structuralsimilarity index measure (SSIM) [Wang et al., 2003] as a basis of our evaluation, which is the mostcommon evaluation metric in medical image reconstruction.

Original: In all of our experiments, we follow the setup of the fastMRImulti-coil knee track with 8 × acceleration in order to provide comparison with state-of-the-artnetworks of the Public Leaderboard.
Revised: In all of our experiments, we follow thesetup of the fastMRI multi-coil knee track with 8 × acceleration in order to provide comparison withstate-of-the-art networks of the Public Leaderboard.

Original: That is, we perform retrospective undersampling of the fullysampled k-space data by randomly subsampling 12 .
Revised: That is, we perform retrospective undersamplingof the fully-sampled k-space data by randomly subsampling 12 .

Original: 5% of whole k-space lines in the phase encodingdirection, keeping 4% of lowest frequency adjacent k-space lines.
Revised: 5% of whole k-space lines in thephase encoding direction, keeping 4% of lowest frequency adjacent k-space lines.

Original: During training, we generaterandom masks following the above method, whereas for the validation dataset we keep the masksﬁxed for each k-space volume.
Revised: During training, wegenerate random masks following the above method, whereas for the validation dataset we keep themasks ﬁxed for each k-space volume.

Revised: For HUMUS-Net, we center crop and pad inputs to 384 × 384 .

Original: We compare the reconstruction quality of our proposed model withthe current best performing network, E2E-VarNet.
Revised: We compare the reconstruction quality of our proposed model with the current best performingnetwork, E2E-VarNet.

Original: For details on HUMUS-Net hyperparameters andtraining, we refer the reader to the supplementary.
Revised: For details on HUMUS-Net hyperparameters and training, we refer the readerto the supplementary.

Original: For E2E-VarNet, we use the hyperparametersspeciﬁed in Sriram et al.
Revised: For E2E-VarNet, we use the hyperparameters speciﬁed in Sriram et al.

Original: fastMRI–
Revised: fastMRI

Original:  The fastMRI dataset [Zbontar et al., 2019] is the largest publicly available MRI datasetwith competitive baseline models and a public leaderboard, and thus provides an opportunity todirectly compare different algorithms.
Revised: – The fastMRI dataset [Zbontar et al., 2019] is the largest publicly available MRI datasetwith competitive baseline models and a public leaderboard, and thus provides an opportunity todirectly compare different algorithms.

Original: Speciﬁcally, we run experiments on the multi-coil knee dataset,consisting of close to 35 k slices in 973 volumes.
Revised: Speciﬁcally, we run experiments on the multi-coil knee dataset,consisting of close to 35 k slices in 973 volumes.

Original: We use the default HUMUS-Net model deﬁnedabove with 3 adjacent slices as input. Table 1 demonstrates our results compared to the best publishedmodels from the fastMRI Leaderboard 2 evaluated on the test dataset. Our model establishes new stateof the art in terms of SSIM on this dataset by a large margin, and achieves comparable or slightly
Revised: We use the default HUMUS-Net model deﬁned

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: better performance than other methods in terms of PSNR and NMSE.
Revised: above with 3 adjacent slices as input. Table 1 demonstrates our results compared to the best publishedmodels from the fastMRI Leaderboard 2 evaluated on the test dataset. Our model establishes new stateof the art in terms of SSIM on this dataset by a large margin, and achieves comparable or slightlybetter performance than other methods in terms of PSNR and NMSE.

Original: Moreover, as seen in the secondcolumn of Table 2, we evaluated our model on the fastMRI validation dataset as well and comparedour results to E2E-VarNet, the best performing model from the leaderboard.
Revised: Moreover, as seen in the secondcolumn of Table 2, we evaluated our model on the fastMRI validation dataset as well and comparedour results to E2E-VarNet, the best performing model from the leaderboard.

Original: We observe similarimprovements in terms of the reconstruction SSIM metric to the test dataset.
Revised: We observe similarimprovements in terms of the reconstruction SSIM metric to the test dataset.

Original: Visual inspection ofreconstructions shows that HUMUS-Net recovers very ﬁne details in images that may be missed byother state-of-the-art reconstruction algorithms (see attached ﬁgures in the supplementary).
Revised: Visual inspection ofreconstructions shows that HUMUS-Net recovers very ﬁne details in images that may be missed byother state-of-the-art reconstruction algorithms (see attached ﬁgures in the supplementary).

Original: Stanford 2D – Next, we run experiments on the Stanford2D FSE [Cheng] dataset, a publicly available
Revised: Stanford 2D – Next, we run experiments on the Stanford2D FSE [Cheng] dataset, a publicly available

Original: MRI dataset consisting of scans from various anatomies (pelvis, lower extremity and more) in 89fully-sampled volumes.
Revised: MRI dataset consisting of scans from various anatomies (pelvis, lower extremity and more) in 89fully-sampled volumes.

Original: We randomly sample 80% of volumes as train data and use the rest forvalidation.
Revised: We randomly sample 80% of volumes as train data and use the rest forvalidation.

Original: We randomly generate 3 different train-validation splits this way to reduce variations in thepresented metrics.
Revised: We randomly generate 3 different train-validation splits this way to reduce variations in thepresented metrics.

Original: As slices in this dataset have widely varying shapes across volumes, we center cropthe target images to keep spatial resolution within 384 × 384 .
Revised: As slices in this dataset have widely varying shapes across volumes, we center cropthe target images to keep spatial resolution within 384 × 384 .

Original: We use the default HUMUS-Net deﬁnedabove with single slices as input.
Revised: We use the default HUMUS-Net deﬁnedabove with single slices as input.

Original: Our results comparing the best performing MRI reconstructionmodel with HUMUS-Net is shown in the third column of Table 2.
Revised: Our results comparing the best performing MRI reconstructionmodel with HUMUS-Net is shown in the third column of Table 2.

Original: We present the mean SSIM ofall runs along with the standard error.
Revised: We present the mean SSIM ofall runs along with the standard error.

Original: We achieve improvements of similar magnitude as on thefastMRI dataset.
Revised: We achieve improvements of similar magnitude as on thefastMRI dataset.

Original: These results demonstrate the effectiveness of HUMUS-Net on a more diversedataset featuring multiple anatomies.
Revised: These results demonstrate the effectiveness of HUMUS-Net on a more diversedataset featuring multiple anatomies.

Original: Stanford 3D – Finally, we evaluate our model on the Stanford Fullysampled 3D FSE Knees dataset[Sawyer et al., 2013], a public MRI dataset including 20 volumes of knee MRI scans.
Revised: Stanford 3D – Finally, we evaluate our model on the Stanford Fullysampled 3D FSE Knees dataset[Sawyer et al., 2013], a public MRI dataset including 20 volumes of knee MRI scans.

Original: We generatetrain-validation splits using the method described for Stanford 2D and perform 3 runs.
Revised: We generatetrain-validation splits using the method described for Stanford 2D and perform 3 runs.

Original: We use thedefault HUMUS-Net network with single slices as input.
Revised: We use thedefault HUMUS-Net network with single slices as input.

Original: The last column of Table 2 comparesour results to E2E-VarNet, showing improvements of similar scales as on other datasets we haveinvestigated in this work.
Revised: The last column of Table 2 comparesour results to E2E-VarNet, showing improvements of similar scales as on other datasets we haveinvestigated in this work.

Original: This experiment demonstrates that HUMUS-Net performs well not only onlarge-scale MRI datasets, but also on smaller problems.
Revised: This experiment demonstrates that HUMUS-Net performs well not only onlarge-scale MRI datasets, but also on smaller problems.

Original: 5.2 Ablation Studies
Revised: 5.2 Ablation Studies

Original: In this section we motivate our design choices by ablating different parts of our network and compareit to our proposed architecture.
Revised: In this section, we motivate our design choices through a set of ablation studies. We start from SwinIR, a general image reconstruction network, and highlight its weaknesses for MRI. Then, wedemonstrate step-by-step how we addressed these shortcomings and arrived at the HUMUS-Netarchitecture.

Original: In all experiments, we train the models on the Stanford 3D dataset.
Revised: We train the models on the Stanford 3D dataset.

Original: Training and hyperparameter details can be found in the supplementary.
Revised: More details can be found in thesupplementary.

Original: The results of our ablationstudies are summarized in Table [-
-]
Revised: The results of our ablation studies are summarized in Table [-
-]3.

Original: First, we investigate SwinIR, a state-of-the-art image denoiser and super-resolution model thatfeatures a hybrid Transformer-convolutional architecture.
Revised: First, we investigate SwinIR, a state-of-the-art image denoiser and super-resolution model thatfeatures a hybrid Transformer-convolutional architecture.

Original: In order to handle the 10 × larger inputsizes ( 320 × 320 ) in our MRI dataset compared to the input images this network has been designedfor ( 128 × 128 ), we reduce the embedding dimension of SwinIR to ﬁt into GPU memory (16 GB).
Revised: In order to handle the 10 × larger inputsizes ( 320 × 320 ) in our MRI dataset compared to the input images this network has been designedfor ( 128 × 128 ), we reduce the embedding dimension of SwinIR to ﬁt into GPU memory (16 GB).

Original: We ﬁnd that compared to models designed for MRI reconstruction, such as E2E-VarNet (last row in
Revised: We ﬁnd that compared to models designed for MRI reconstruction, such as E2E-VarNet (last row in

Original: Table 3) SwinIR performs poorly.
Revised: Table 3) SwinIR performs poorly.

Original: This is not only due to the reduced network size, but also due tothe fact that SwinIR is not speciﬁcally designed to take the MRI forward model into consideration.
Revised: This is not only due to the reduced network size, but also due tothe fact that SwinIR is not speciﬁcally designed to take the MRI forward model into consideration.

Original: Next, we unroll SwinIR and add a sensitivity map estimator similarly to the architecture of HUMUSNet and E2E-VarNet.
Revised: Next, we unroll SwinIR and add a sensitivity map estimator.

Original: We refer to this model as Un-SS .
Revised: We refer to this model as Un-SS .

Original: Due to unrolling, we have to further reducethe embedding dimension of the denoiser and also decrease the depth of the network in order toﬁt into GPU memory.
Revised: Due tounrolling, we have to further reduce the embedding dimension of the denoiser and also decrease thedepth of the network in order to ﬁt into GPU memory.

Original: Un-SS , due to its small size, performs slightly worse than vanilla SwinIRand signiﬁcantly lags behind the E2E-VarNet architectures.
Revised: Un-SS , due to its small size, performs slightlyworse than vanilla SwinIR and signiﬁcantly lags behind the E2E-VarNet architectures.

Original: We note that SwinIR operates over asingle, full-resolution scale, whereas state-of-the-art MRI reconstruction models typically incorporatemulti-scale processing in the form of U-Net-like architectures.
Revised: We note that SwinIR operates over a single, full-resolution scale, whereas state-of-the-art MRI reconstructionmodels typically incorporate multi-scale processing in the form of U-Net-like architectures.

Original: Thus, we replace SwinIR by MUST, our proposed multi-scale hybrid processing unit, but keep theembedding dimension in the largest-resolution scale ﬁxed.
Revised: Thus, we replace SwinIR by MUST, our proposed multi-scale hybrid processing unit, but keep theembedding dimension in the largest-resolution scale ﬁxed.

Original: The obtained network, which we call
Revised: The obtained network, which we call

Original: Un-MS , has overall lower computational cost when compared with Un-SS , however as Table 3 shows
Revised: Un-MS , has overall lower computational cost when compared with Un-SS , however as Table 3 shows

Original: MRI reconstruction performance has signiﬁcantly improved compared to both Un-SS and vanilla
Revised: MRI reconstruction performance has signiﬁcantly improved compared to both Un-SS and vanilla

Original: SwinIR, which highlights the efﬁciency of our proposed multi-scale feature extractor.
Revised: SwinIR, which highlights the efﬁciency of our proposed multi-scale feature extractor.

Original: Reconstructionperformance is limited by the low dimension of patch embeddings, which we are unable to increasefurther due to our compute and memory constraints originating in the high-resolution inputs.
Revised: Reconstructionperformance is limited by the low dimension of patch embeddings, which we are unable to increasefurther due to our compute and memory constraints originating in the high-resolution inputs.

Original: The most straightforward approach to tackle the challenge of high input resolution is to increase thepatch size.
Revised: The most straightforward approach to tackle the challenge of high input resolution is to increase thepatch size.

Original: To test this idea, we take Un-MS and embed 2 × 2 patches of the inputs, thus reducing thenumber of tokens processed by the network by a factor of 4 .
Revised: To test this idea, we take Un-MS and embed 2 × 2 patches of the inputs, thus reducing thenumber of tokens processed by the network by a factor of 4 .

Original: We refer to this model as Un-MS-Patch2 .
Revised: We refer to this model as Un-MS-Patch2 .

Original: This reduction in compute and memory load allows us to increase network capacity by increasing theembedding dimension 3 -folds (to ﬁll GPU memory again).
Revised: This reduction in compute and memory load allows us to increase network capacity by increasing theembedding dimension 3 -folds (to ﬁll GPU memory again).

Original: However, Un-MS-Patch2 performs muchworse than previous models using patch size of 1 .
Revised: However, Un-MS-Patch2 performs muchworse than previous models using patch size of 1 .

Original: For classiﬁcation problems, where the generalimage context is more important than small details, patches of 16 × 16 or 8 × 8 are considered typical[Dosovitskiy et al., 2020].
Revised: For classiﬁcation problems, where the generalimage context is more important than small details, patches of 16 × 16 or 8 × 8 are considered typical[Dosovitskiy et al., 2020].

Original: Even for more dense prediction tasks such as medical image segmentation,patch size of 4 × 4 have been used successfully Cao et al.
Revised: Even for more dense prediction tasks such as medical image segmentation,patch size of 4 × 4 have been used successfully Cao et al.

Original: [2021a].
Revised: [2021a].

Original: However, our experiments suggestthat in low-level vision tasks such as MRI reconstruction using patches larger than 1 × 1 may bedetrimental due to loss of crucial high-frequency detail information.
Revised: However, our experiments suggestthat in low-level vision tasks such as MRI reconstruction using patches larger than 1 × 1 may bedetrimental due to loss of crucial high-frequency detail information.

Original: Our approach to address the heavy computational load of Transformers for large input resolutionswhere increasing the patch size is not an option is to process lower resolution features extracted viaconvolutions.
Revised: Our approach to address the heavy computational load of Transformers for large input resolutionswhere increasing the patch size is not an option is to process lower resolution features extracted viaconvolutions.

Original: That is we replace MUST in Un-MS by a HUMUS-Block, resulting in our proposed
Revised: That is we replace MUST in Un-MS by a HUMUS-Block, resulting in our proposed

Original: HUMUS-Net architecture.
Revised: HUMUS-Net architecture.

Original: We train a smaller version of HUMUS-Net with the same embeddingdimension as Un-MS .
Revised: We train a smaller version of HUMUS-Net with the same embeddingdimension as Un-MS .

Original: As seen in Table 3, our model achieves the best performance across allother proposed solutions, even surpassing E2E-VarNet (even though we have not optimized allarchitectural hyperparameters for HUMUS-Net in this experiment, such as the optimal number ofunrolled iterations).
Revised: As seen in Table 3, our model achieves the best performance across all otherproposed solutions, even surpassing E2E-VarNet.

Original: This series of incremental studies highlights the importance of each architecturaldesign choice leading to our proposed HUMUS-Net architecture.
Revised: This series of incremental studies highlights theimportance of each architectural design choice leading to our proposed HUMUS-Net architecture.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce HUMUS-Net, an unrolled, Transformer-convolutional hybrid networkfor accelerated MRI reconstruction.
Revised: In this paper, we introduce HUMUS-Net, an unrolled, Transformer-convolutional hybrid networkfor accelerated MRI reconstruction.

Original: HUMUS-Net achieves state-of-the-art performance on thefastMRI dataset and greatly outperforms all previous published and reproducible methods.
Revised: HUMUS-Net achieves state-of-the-art performance on thefastMRI dataset and greatly outperforms all previous published and reproducible methods.

Original: Wedemonstrate the performance of our proposed method on two other MRI datasets and perform ﬁnegrained ablation studies to motivate our design choices and emphasize the compute and memorychallenges of Transformer-based architectures on low-level and dense computer vision tasks suchas MRI reconstruction.
Revised: Wedemonstrate the performance of our proposed method on two other MRI datasets and perform ﬁnegrained ablation studies to motivate our design choices and emphasize the compute and memorychallenges of Transformer-based architectures on low-level and dense computer vision tasks suchas MRI reconstruction.

Original: A limitation of our current architecture is that it requires ﬁxed-size inputs,which we intend to address with a more ﬂexible design in the future.
Revised: A limitation of our current architecture is that it requires ﬁxed-size inputs,which we intend to address with a more ﬂexible design in the future.

Original: This work opens the door for theadoption of a multitude of promising techniques introduced recently in the literature for Transformers,which we leave for future work.
Revised: This work opens the door for theadoption of a multitude of promising techniques introduced recently in the literature for Transformers,which we leave for future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Conclusions.
Revised: [Yes] See Conclusions.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We were
Revised: [No] We were

Original: unable to identify possible negative effects on society.
Revised: unable to identify possible negative effects on society.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Full code is
Revised: [Yes] Full code is

Original: available in the supplementary.
Revised: available in the supplementary.

Original: Source code will be published upon paper acceptance.
Revised: Source code will be published upon paper acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Supplementary
Revised: [Yes] See Supplementary

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report the standard error of runs in our tables.
Revised: [Yes] We report the standard error of runs in our tables.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Supplementary.
Revised: [Yes] See Supplementary.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]We attach source code in the Supplementary.
Revised: [Yes] We attach source code in the Supplementary.

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re[Table]
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable[Table]
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table]
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: AppendixA HUMUS-Net baseline details

Revised: Our default model has 3 RSTB-D downsampling blocks, 2 RSTB-B bottleneck blocks and 3 RSTB-Uupsampling blocks with 3 − 6 − 12 attention heads in the D/U blocks and 24 attention heads inthe bottleneck block.

Revised: For Swin Transformers layers, the window size is 8 for all methods and MLPratio ( hidden _ dim/input _ dim ) of 2 is used.

Revised: Each RSTB block consists of 2 STLs with embeddingdimension of 66 .

Revised: We use 8 cascades of unrolling with a U-Net as sensitivity map estimator (same asin E2E-VarNet) with 16 channels.

Revised: We center crop and reﬂection pad the input images to 384 × 384 resolution for HUMUS-Net and use the complete images for VarNet.

Revised: In all experiments, we minimize the SSIM loss between the target image x ∗ and the reconstruction ˆ x deﬁned as

Revised: [Equation]

Revised: fastMRI experiments– We train HUMUS-Net using Adam for 50 epochs with a learning rate of0 . 0001 , dropped by a factor of 10 at epoch 40 and apply adjacent slice reconstruction with 3 slices.

Revised: We run the experiments on 8 × Titan RTX 5000 24GB GPUs with a per-GPU batch size of 1 .

Revised: Stanford 3D experiments– We train for 25 epochs with a learning rate of 0 . 0001 , dropped by afactor of 10 at epoch 20 and reconstruct single slices.

Revised: We run the experiments on 4 × Quadro RTX5000 16GB GPUs with a per-GPU batch size of 1 .

Revised: Stanford 2D experiments– We train for 50 epochs with a learning rate of 0 . 0001 , dropped by afactor of 10 at epoch 40 and reconstruct single slices.

Revised: Moreover, we crop reconstruction targets to ﬁtinto a 384 × 384 box.

Revised: We run the experiments on 8 × Quadro RTX 5000 16GB GPUs with a per-GPUbatch size of 1 .

Revised: B Ablation study experimental details

Revised: Here we discuss the experimental setting and hyperparameters used in our ablation study in Section5.2.

Revised: In all experiments, we train the models on the Stanford 3D dataset for 3 different train-validationsplits and report the mean and standard error of the results.

Revised: We use Adam optimizer with learningrate 0 . 0001 and train for 25 epochs, decaying the learning rate by a factor of 10 at 20 epochs.

Revised: Forall unrolled methods, we unroll 12 iterations in order to provide direct comparison with the bestperforming E2E-VarNet, which uses the same number of cascades.

Revised: We reconstruct single slices anddo not use the method of adjacent slice reconstruction discussed in 4.4.

Revised: In models with sensitivitymap estimator,we used the default U-Net with 8 channels (same as default E2E-VarNet).

Revised: For Swin

Revised: Transformers layers, the window size is 8 for all methods and MLP ratio ( hidden _ dim/input _ dim )of 2 is used.

Revised: Further hyperparameters are summarized in Table 4.

Revised: [Table]

Revised: We run the experiments on 4 × Quadro RTX 5000 16GB GPUs

Revised: [Figure]

Revised: C Number of unrolled iterations: ablation study

Revised: The number of cascades in unrolled networks has a fundamental impact on their performance.

Revised: Deeper networks typically perform better, but also incur heavy computational and memory cost.

Revised: Inthis experiment we investigate the scaling of HUMUS-Net with respect to the number of iterativeunrollings.

Revised: We perform an ablation study on the Stanford 3D dataset with × 8 acceleration for a ﬁxed trainingvalidation split, where 20% of the training set has been set aside for validation.

Revised: We reconstruct singleslices, without applying adjacent slice reconstruction.

Revised: We plot the highest SSIM throughout trainingon the validation dataset for each network on Figure 6.

Revised: We observe improvements in reconstruction performance with increasing number of cascades, andthis improvement has not saturated yet in the range of model sizes we have investigated.

Revised: In ourmain experiments, we select 8 cascades due to memory and compute limitations for deeper models.

Revised: However, our experiment suggests that HUMUS-Net can potentially obtain even better reconstructionresults given enough computational resources.

Revised: D Adjacent slice reconstruction: ablation study

Revised: In this section, we demonstrate that even though adjacent slice reconstruction is in general helpful, itis not the main reason why HUMUS-Net performs better than E2E-VarNet.

Revised: To this end, we add adjacent slice reconstruction to E2E-VarNet and investigate its effect on modelperformance.

Revised: We run experiments on the Stanford 3D dataset for 3 different random train-validationsplits.

Revised: We use adjacent slice reconstruction with 3 slices.

Revised: We have found that the default learningrate for E2E-VarNet is not optimal with the increased input size, therefore we tune the learning rateusing grid search and set it to 0 . 0005 .

Revised: We compare the results with HUMUS-Net, where we matchthe number of cascades in E2E-VarNet, and use the default embedding dimension of 66 .

Revised: We do notuse adjacent slice reconstruction when training HUMUS-Net in order to ablate its effect.

Revised: The results are summarized in Table 5.

Revised: We observe that ASR boosts the reconstruction quality of

Revised: E2E-VarNet.

Revised: However, ASR alone cannot close the gap between the two models, as HUMUS-Netwithout ASR still outperforms the best E2E-VarNet model with ASR.

Original: [Table]
Revised: [Table]

Revised: E Iterative denoising visualization

Revised: Here, we provide more discussion on the iterative denoising interpretation of unrolled networks,such as HUMUS-Net.

Revised: Consider the regularized inverse problem formulation of MRI reconstructionin (2.1) and design the architecture based on unrolling the gradient descent steps on the aboveproblem.

Revised: Applying data consistency corresponds to the gradient of the ﬁrst term in (2.2), whereas welearn the gradient of the regularizer parameterized by the HUMUS-Block architecture.

Revised: This can beconceptualized as alternating between enforcing consistency with the measurements and applying adenoiser based on some prior, represented by the HUMUS-Block in our architecture.

Revised: We show an example of a sequence of intermediate reconstructions in order to support the denoisingintuition in Figure 7.

Revised: We plot the magnitude image of reconstructions at the outputs of consecutive

Revised: HUMUS-Blocks, along with the model input zero-padded reconstruction.

Revised: We point out that in generalthe intermediate reconstruction quality (for instance in terms of SSIM) is not necessarily an increasingfunction of cascades.

Revised: This is due to the highly non-linear nature of the mapping represented by theneural network, which might not always be intuitive and interpretable.

Revised:  F Vision Transformer terminology overview

Revised: Here we provide a brief overview of the terms related to the Transformer architecture, some of whichhas been carried over from the Natural Language Processing (NLP) literature.

Revised: We introduce the keyconcepts through the Vision Transformer architecture, which has inspired most Transformer-basedarchitectures in computer vision.

Revised: For more details, we refer the reader to [Dosovitskiy et al., 2020].

Revised: Traditional Transformers for NLP receive a sequence of 1D token embeddings.

Revised: Each token may represent for instance a word in a sentence.

Revised: To extend this idea to 2D images, the input x ∈ R H × W × C image ( H and W stand for spatial dimensions, C for channel dimension) is split into N patches of size P × P , and each patch is ﬂattened into a 1D vector to produce an input of ﬂattened patches x f with shape N × P 2 C : x f =

Revised: PatchFlatten ( x ) .

Revised: In order to set the latent dimension of the input entering the Vision Transformer, a trainable linear mapping E ∈ R P 2 C × D is applied to the ﬂattened patches, resulting in the so called patch embeddings z of shape N × D : z

Revised: [Equation]

Revised: f E.

Revised: In order to encode information with respect to the position of image patches, a learnable positional embedding E pos ∈ R N × D is added to the patch embeddings:

Revised: [Equation]

Revised: The input to the Transformer encoder is this N × D representation, which we also refer to in thepaper as token representation, as each row in the representation corresponds to a token (in our casean image patch) in the original input.

Revised: G Detailed validation results

Revised: [Table]

Revised: H Additional ﬁgures

Revised: [Figure]

Revised: [Equation]

Revised: HUMUS-Net reconstructs ﬁne details on MRI images that other state-of-the-art methods may miss.

Revised: I Results on additional accelerations

Revised: We observe that HUMUS-Net achieves state-of-the-art performance across a wide range of acceleration ratios.

Revised: We perform experiments on the Stanford 3D dataset using a small HUMUS-Net modelwith only 6 cascades, embedding dimension of 66 , adjacent slice reconstruction with 3 slices and for the sake of simplicity we removed the residual path from the HUMUS-Block.

Revised: We call this model

Revised: HUMUS-Net-small .

Revised: We set the learning rate to 0 . 0002 and train the model with Adam optimizer.

Revised: We generate 3 random training-validation splits on the Stanford 3D dataset (same splits as in other

Revised: Stanford 3D experiments in this paper), and show the mean and standard error of the results in Table .

Revised: Acceleration

Revised: [Table]