Original: S CALING L AWS F OR D EEP L EARNING B ASED I MAGE R ECONSTRUCTION
Revised: S CALING L AWS F OR D EEP L EARNING B ASED I MAGE R ECONSTRUCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for a variety of linear inverse problems.
Revised: Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for a variety of linear inverse problems.

Original: Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains.
Revised: Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains.

Original: In this work, we study whether major performance gains are expected from scaling up the training set size.
Revised: In this work, we study whether major performance gains are expected from scaling up the training set size.

Original: We consider image denoising, accelerated magnetic resonance imaging, and super-resolution and empirically determine the reconstruction quality as a function of training set size, while optimally scaling the network size.
Revised: We consider image denoising, accelerated magnetic resonance imaging, and super-resolution and empirically determine the reconstruction quality as a function of training set size, while simultaneously scaling the network size.

Original: For all three tasks we find that an initially steep power-law scaling slows significantly already at moderate training set sizes.
Revised: For all three tasks we find that an initially steep powerlaw scaling slows significantly already at moderate training set sizes.

Original: Interpolating those scaling laws suggests that even training on millions of images would not significantly improve performance.
Revised: Interpolating those scaling laws suggests that even training on millions of images would not significantly improve performance.

Original: To understand the expected behavior, we analytically characterize the performance of a linear estimator learned with early stopped gradient descent.
Revised: To understand the expected behavior, we analytically characterize the performance of a linear estimator learned with early stopped gradient descent.

Original: The result formalizes the intuition that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.
Revised: The result formalizes the intuition that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks trained to map a noisy measurement of an image or a noisy image to a clean image give state-of-the-art (SOTA) performance for image reconstruction problems.
Revised: Deep neural networks trained to map a noisy measurement of an image or a noisy image to a clean image give state-of-the-art (SOTA) performance for image reconstruction problems.

Original: Examples are image denoising (Burger et al., 2012; Zhang et al., 2017;
Revised: Examples are image denoising (Burger et al., 2012; Zhang et al., 2017;

Original: Brooks et al., 2019; Liang et al., 2021), super-resolution (Dong et al., 2016; Ledig et al., 2017; Liang et al., 2021), and compressive sensing for computed tomography (CT) (Jin et al., 2017), and accelerated magnetic resonance imaging (MRI)
Revised: Brooks et al., 2019; Liang et al., 2021), super-resolution (Dong et al., 2016; Ledig et al., 2017; Liang et al., 2021), and compressive sensing for computed tomography (CT) (Jin et al., 2017), and accelerated magnetic resonance imaging (MRI)

Original: (Zbontar et al., 2018; Sriram et al., 2020; Muckley et al., 2021; Fabian & Soltanolkotabi, 2022).
Revised: (Zbontar et al., 2018; Sriram et al., 2020; Muckley et al., 2021; Fabian & Soltanolkotabi, 2022).

Original: The performance of a neural network for imaging is determined by the network architecture and optimization, the size of the network, and the size and quality of the training set.
Revised: The performance of a neural network for imaging is determined by the network architecture and optimization, the size of the network, and the size and quality of the training set.

Original: Significant work has been invested in architecture development.
Revised: Significant work has been invested in architecture development.

Original: For example, in the field of accelerated MRI, networks started out as convolutional neural networks (CNN) (Wang et al., 2016), which are now often used as building blocks in un-rolled variational networks (Hammernik et al., 2018;
Revised: For example, in the field of accelerated MRI, networks started out as convolutional neural networks (CNN) (Wang et al., 2016), which are now often used as building blocks in un-rolled variational networks (Hammernik et al., 2018;

Original: Sriram et al., 2020).
Revised: Sriram et al., 2020).

Original: Most recently, transformers have been adapted to image reconstruction (Lin &
Revised: Most recently, transformers have been adapted to image reconstruction (Lin &

Original: Heckel, 2021; Huang et al., 2022; Fabian & Soltanolkotabi, 2022).
Revised: Heckel, 2022; Huang et al., 2022; Fabian & Soltanolkotabi, 2022).

Original: However, it is not clear how substantial the latest improvements through architecture design are compared to potential improvements expected by scaling the training set and network size.
Revised: However, it is not clear how substantial the latest improvements through architecture design are compared to potential improvements expected by scaling the training set and network size.

Original: Contrary to natural language processing (NLP) models and modern image classifiers that are trained on billions of examples, networks for image reconstruction are only trained on hundreds to thousands of example images.
Revised: Contrary to natural language processing (NLP) models and modern image classifiers that are trained on billions of examples, networks for image reconstruction are only trained on hundreds to thousands of example images.

Original: For example, the training set of SwinIR (Liang et al., 2021), the current SOTA for image denoising, contains only 10k images, and the popular benchmark dataset for accelerated MRI consists only of 35k images (Zbontar et al., 2018).
Revised: For example, the training set of SwinIR (Liang et al., 2021), the current SOTA for image denoising, contains only 10k images, and the popular benchmark dataset for accelerated MRI consists only of 35k images (Zbontar et al., 2018).

Original: In this work, we study whether neural networks for image reconstruction only require moderate amounts of data to reach their peak performance, or whether major boosts are expected from increasing the training set size.
Revised: In this work, we study whether neural networks for image reconstruction only require moderate amounts of data to reach their peak performance, or whether major boosts are expected from increasing the training set size.

Original: To partially address this question, we focus on three problems: image denoising, reconstruction from few and noisy measurements (compressive sensing) in the context of accelerated
Revised: To partially address this question, we focus on three problems: image denoising, reconstruction from few and noisy measurements (compressive sensing) in the context of accelerated

Original: MRI, and super-resolution.
Revised: MRI, and super-resolution.

Original: We pick Gaussian denoising for its practical importance and since it can
Revised: We pick Gaussian denoising for its practical importance and since it can

Original: [Figure]
Revised: [Figure]

Original: serve as a building block to solve more general image reconstruction problems well (Venkatakrishnan et al., 2013).
Revised: serve as a building block to solve more general image reconstruction problems well (Venkatakrishnan et al., 2013).

Original: We pick MR reconstruction because it is an important instance of a compressive sensing problem, and many problems can be formulated as compressive sensing problems, for example super-resolution and in-painting.
Revised: We pick MR reconstruction because it is an important instance of a compressive sensing problem, and many problems can be formulated as compressive sensing problems, for example super-resolution and in-painting.

Original: In addition, for MR reconstruction the question on how much data is needed is particularly important, since it is expensive to collect medical data.
Revised: In addition, for MR reconstruction the question on how much data is needed is particularly important, since it is expensive to collect medical data.

Original: For the thee problems we identify scaling laws that describe the reconstruction quality as a function of thenumber of training examples, for optimally scaled network sizes.
Revised: For the three problems we identify scaling laws that describe the reconstruction quality as a function of the training set size, while simultaneously scaling network sizes.

Original: Such scaling laws have been established for NLP and classification tasks before, as discussed below, but not for image reconstruction.
Revised: Such scaling laws have been established for NLP and classification tasks, as discussed below, but not for image reconstruction.

Original: The experiments are conducted with a U-Net (Ronneberger et al., 2015) and the SOTA SwinIR
Revised: The experiments are conducted with a U-Net (Ronneberger et al., 2015) and the SOTA SwinIR

Original: (Liang et al., 2021), a transformer architecture.
Revised: (Liang et al., 2021), a transformer architecture.

Original: We primarily consider the U-net since it is is widely used for image reconstruction and acts as a building block in SOTA models for image denoising (Brooks et al., 2019; Gurrola-Ramos et al., 2021; Zhang et al., 2021; 2022) and accelerated MRI (Zbontar et al., 2018; Sriram et al., 2020).
Revised: We primarily consider the U-Net since it is is widely used for image reconstruction and acts as a building block in SOTA models for image denoising (Brooks et al., 2019; Gurrola-Ramos et al., 2021; Zhang et al., 2021; 2022) and accelerated MRI (Zbontar et al., 2018; Sriram et al., 2020).

Original: We also present results for denoising with the SOTA SwinIR
Revised: We also present results for denoising with the SwinIR.

Original: (Liang et al., 2021), a transformer architecture.

Original: The SwinIR outperforms the U-Net, but we find that its scaling with the number of training examples does not differ notably.
Revised: The SwinIR outperforms the U-Net, but we find that its scaling with the number of training examples does not differ notably.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • Empirical scaling laws for denoising.
Revised: • Empirical scaling laws for denoising.

Original: We train U-Nets of sizes 0.1M to 46.5M parameters with training set sizes from 100 to 100k images from the ImageNet dataset (Russakovsky et al., 2015) for Gaussian denoising with 20 .
Revised: We train U-Nets of sizes 0.1M to 46.5M parameters with training set sizes from 100 to 100k images from the ImageNet dataset (Russakovsky et al., 2015) for Gaussian denoising with 20 .

Original: 17 dB Peak-Signal-to-Noise ratio (PSNR).
Revised: 17 dB Peak-Signal-to-Noise ratio (PSNR).

Original: While for the largest training set sizes and network sizes we consider, performance continues to increase, the rate of performance increase for training set sizes beyond a few thousand images slows to a level indicating that even training on millions of images only yields a marginal benefit, see Fig. 1(a).
Revised: While for the largest training set sizes and network sizes we consider, performance continues to increase, the rate of performance increase for training set sizes beyond a few thousand images slows to a level indicating that even training on millions of images only yields a marginal benefit, see Fig. 1(a).

Original: We also train SOTA SwinIRs of sizes 4M to 129M parameters on the same range of training set sizes, see Fig. 2(a).
Revised: We also train SOTA SwinIRs of sizes 4M to 129M parameters on the same range of training set sizes, see Fig. 2(a).

Original: Albeit it performs better than the U-net, its scaling behavior is essentially equivalent, and again after a few thousand images, onlyvery marginal benefits are expected.
Revised: Albeit it performs better than the U-Net, its scaling behavior is essentially equivalent, and again after a few thousand images, only marginal benefits are expected.

Original: Scaling up its training set and network size did, however, give benefits, the largest model we trained yields new SOTA results on four common test sets by 0.05 to 0.21dB. • Empirical scaling laws for compressive sensing.
Revised: Scaling up its training set and network size did, however, give benefits, the largest model we trained yields new SOTA results on four common test sets by 0.05 to 0.22dB. • Empirical scaling laws for compressive sensing.

Original: We train U-nets of sizes from 2M to 500M parameters for 4x accelerated MRI with training set sizes from 50 to 50k images from the fastMRI dataset (Zbontar et al., 2018).
Revised: We train U-Nets of sizes from 2M to 500M parameters for 4x accelerated MRI with training set sizes from 50 to 50k images from the fastMRI dataset (Knoll et al., 2020b).

Original: We again find that beyond a dataset size of about 2.5k, the rate of improvement as a function of the training set size considerably slows, see Fig. 1(c).
Revised: We again find that beyond a dataset size of about 2.5k, the rate of improvement as a function of the training set size considerably slows, see Fig. 1(c).

Original: This indicates that while models are expected to improve by increasing the training set size, we expect that training on millions of images does not significantly improves performance.
Revised: This indicates that while models are expected to improve by increasing the training set size, we expect that training on millions of images does not significantly improves performance.

Original: • Empirical scaling laws from super-resolution .
Revised: • Empirical scaling laws for super-resolution .

Original: We also briefly study super-resolution, and similarly as for denoising and compressive sensing find a slowing of the scaling law at moderate dataset sizes.
Revised: We also briefly study super-resolution, and similarly as for denoising and compressive sensing find a slowing of the scaling law at moderate dataset sizes.

Original: See appendix C. • Understanding scaling laws for denoising theoretically.
Revised: See appendix C. • Understanding scaling laws for denoising theoretically.

Original: Our empirical results indicate that the denoising performance of a neural network trained end-to-end doesn’t increase as a function of training examples beyond a certain point.
Revised: Our empirical results indicate that the denoising performance of a neural network trained end-to-end doesn’t increase as a function of training examples beyond a certain point.

Original: This is expected since once we reach a noise specific error floor, more data is not beneficial.
Revised: This is expected since once we reach a noise specific error floor, more data is not beneficial.

Original: We make this intuition precise for a linear estimator learned with early stopped gradient descent to denoise data drawn from a d -dimensional linear subspace.
Revised: We make this intuition precise for a linear estimator learned with early stopped gradient descent to denoise data drawn from a d -dimensional linear subspace.

Original: We show that the reconstruction error is upper bounded by d/N plus a noise-dependent error floor level.
Revised: We show that the reconstruction error is upper bounded by d/N plus a noise-dependent error floor level.

Original: Once the error induced by learning the signal model, d/N , is small relative to the error floor, more training examples N are not beneficial.
Revised: Once the error induced by learning the signal model, d/N , is small relative to the error floor, more training examples N are not beneficial.

Original: Together, our empirical results show that neural networks for denoising, compressive sensing, and super-resolution applied in typical setups (i e, Gaussian denoising with 20 . 17 dB PSNR and multi-coil accelerated MRI with 4x acceleration) already operate in a regime where the scaling laws for the training set size are slowing significantly and thus even very large increases of the training data are not expected to improve performance substantially.
Revised: Together, our empirical results show that neural networks for denoising, compressive sensing, and super-resolution applied in typical setups (i.e, Gaussian denoising with 20 . 17 dB PSNR, and multi-coil accelerated MRI with 4x acceleration) already operate in a regime where the scaling laws for the training set size are slowing significantly and thus even very large increases of the training data are not expected to improve performance substantially.

Original: Even relatively modest model improvements such as those obtained by transformers over convolutional networks are larger than what we expect from scaling the number of training examples from tens of thousands to millions.
Revised: Even relatively modest model improvements such as those obtained by transformers over convolutional networks are larger than what we expect from scaling the number of training examples from tens of thousands to millions.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Scaling laws for prediction problems.
Revised: Scaling laws for prediction problems.

Original: Under the umbrella of statistical learning theory convergence rates of 1 /N or 1 / √ N have been established for a range of relatively simple models and distributions, see e.g. Wainwright (2019) for an overview.
Revised: Under the umbrella of statistical learning theory convergence rates of 1 /N or 1 / √ N have been established for a range of relatively simple models and distributions, see e.g. Wainwright (2019) for an overview.

Original: For deep neural networks used in practice, a recent line of work has empirically characterized the performance as a function of training set size and/or network size for classification and NLP (Hestness et al., 2017; Rosenfeld et al., 2019;
Revised: For deep neural networks used in practice, a recent line of work has empirically characterized the performance as a function of training set size and/or network size for classification and NLP (Hestness et al., 2017; Rosenfeld et al., 2019;

Original: Kaplan et al., 2020; Bahri et al., 2021; Zhai et al., 2021; Ghorbani et al., 2021; Bansal et al., 2022).
Revised: Kaplan et al., 2020; Bahri et al., 2021; Zhai et al., 2021; Ghorbani et al., 2021; Bansal et al., 2022).

Original: In those domains, the scaling laws persist even for very large datasets, as described in more detail below.
Revised: In those domains, the scaling laws persist even for very large datasets, as described in more detail below.

Original: In contrast, for image reconstruction, we find that the power-law behavior already slows considerably at relatively small numbers of training examples.
Revised: In contrast, for image reconstruction, we find that the power-law behavior already slows considerably at relatively small numbers of training examples.

Original: Rosenfeld et al.
Revised: Rosenfeld et al.

Original: (2019) find power-law scaling of performance with training set and network size across models and datasets for language modeling and classification.
Revised: (2019) find power-law scaling of performance with training set and network size across models and datasets for language modeling and classification.

Original: However, because the work fixes either training set or network size, while scaling the other, the scaling laws span only moderate ranges before saturating at a level determined by the fixed quantity and not the problem specific error floor.
Revised: However, because the work fixes either training set or network size, while scaling the other, the scaling laws span only moderate ranges before saturating at a level determined by the fixed quantity and not the problem specific error floor.

Original: The papers Kaplan et al.
Revised: The papers Kaplan et al.

Original: Bahri et al.
Revised: Bahri et al.

Original: ; Zhai et al.
Revised: ; Zhai et al.

Original: ; Ghorbani et al.
Revised: ; Ghorbani et al.

Original: Bansal et al.
Revised: Bansal et al.

Original: (2022) including ours scale the dataset size and model size simultaneously, resulting in an improved predictive power of the obtained scaling laws.
Revised: (2022) including ours scale the dataset size and model size simultaneously, resulting in an improved predictive power of the obtained scaling laws.

Original: Hestness et al.
Revised: Hestness et al.

Original: (2017) study models for language and image classification, and attribute deviations from a power-law curve to a lack of fine-tuning the hyperparameters of very large networks.
Revised: (2017) study models for language and image classification, and attribute deviations from a power-law curve to a lack of fine-tuning the hyperparameters of very large networks.

Original: For transformer language models Kaplan et al.
Revised: For transformer language models Kaplan et al.

Original: (2020) find no deviation from a power-law for up to a training set and network size of 1B images and parameters.
Revised: (2020) find no deviation from a power-law for up to a training set and network size of 1B images and parameters.

Original: Further, Zhai et al.
Revised: Further, Zhai et al.

Original: (2021) find the performance for Vision Transformers (Dosovitskiy et al., 2020) for few-shot image classification to deviate from a power-law curve only at extreme model sizes of 0.3-1B parameters and 3B images.[Figure]
Revised: (2021) find the performance for Vision Transformers (Dosovitskiy et al., 2020) for few-shot image classification to deviate from a power-law curve only at extreme model sizes of 0.3-1B parameters and 3B images.

Original: The role of training set size in inverse problems.For image reconstruction and inverse problems
Revised: The role of training set size in inverse problems.

Original:  in general, we are not aware of work studying scaling laws in a principled manner, covering different problems and model architectures.
Revised: For image reconstruction and inverse problems in general, we are not aware of work studying scaling laws in a principled manner, covering different problems and model architectures.

Original: But, when proposing a new method several works study performance as a function of training set and/or network size.
Revised: But, when proposing a new method several works study performance as a function of training set and/or network size.

Original: However, those studies typically only scale  the parameter of interest while fixing all other parameters, unlike our work, which scales dataset size and network size together, which is important for identifying scaling laws.
Revised: However, those studies typically only scale [Figure] the parameter of interest while fixing all other parameters, unlike our work, which scales dataset size and network size together, which is important for identifying scaling laws.

Original: Below, we review the training set sizes, and if available their scaling properties, used by the recent SOTA in image denoising and accelerated MRI.
Revised: Below, we review the training set sizes, and if available their scaling properties, used by the recent SOTA in image denoising and accelerated MRI.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2017) report that for DnCNN, a standard CNN, using more than 400 distinct images with data augmentation only yields negligible improvements.
Revised: (2017) report that for DnCNN, a standard CNN, using more than 400 distinct images with data augmentation only yields negligible improvements.

Original: Chen et al.
Revised: Chen et al.

Original: pre-train an image processing transformer (IPT) of 115.5M network parameters on ImageNet (1.1M distinct images), and report the performance after fine-tuning to a specific task as a function of the size of the pretraining dataset.
Revised: pre-train an image processing transformer (IPT) of 115.5M network parameters on ImageNet (1.1M distinct images), and report the performance after fine-tuning to a specific task as a function of the size of the pretraining dataset.

Original: IPT’s performance for denoising is surpassed by the latest SOTA in form of the
Revised: IPT’s performance for denoising is surpassed by the latest SOTA in form of the

Original: CNN based DRUnet
Revised: CNN based DRUnet

Original: (Chen et al., 2021), the transformer based SwinIR (Liang et al., 2021) and the
Revised: (Chen et al., 2021), the transformer based SwinIR (Liang et al., 2021) and the

Original: Swin-Conv-Unet (Zhang et al., 2022) a combination of the two.
Revised: Swin-Conv-Unet (Zhang et al., 2022) a combination of the two.

Original: Those models have significantly fewer network parameters and were trained on a training set consisting of only ∼ 10k images, leaving the role of training set size in image denoising open.
Revised: Those models have significantly fewer network parameters and were trained on a training set consisting of only ∼ 10k images, leaving the role of training set size in image denoising open.

Original: The number of available training images for accelerated MRI is limited to few publicly available datasets.
Revised: The number of available training images for accelerated MRI is limited to few publicly available datasets.

Original: Hence, current SOTA (Sriram et al., 2020; Fabian & Soltanolkotabi, 2022) are trained on the largest datasets available (the fastMRI dataset), consisting of 35k images for knee and 70k for brain reconstruction (Zbontar et al., 2018; Muckley et al., 2021).
Revised: Hence, current SOTA (Sriram et al., 2020; Fabian & Soltanolkotabi, 2022) are trained on the largest datasets available (the fastMRI dataset), consisting of 35k images for knee and 70k for brain reconstruction (Zbontar et al., 2018; Muckley et al., 2021).

Original: Adaptations of the MLP-Mixer (Tolstikhin et al., 2021) and the Vision Transformer (Dosovitskiy et al., 2020) to MRI (Mansour et al., 2022; Lin & Heckel, 2021) ablate their performance as a function of the training set size and find that non-CNN based approaches require more data to achieve a similar performance, but potentially also benefit more from even larger datasets.
Revised: Adaptations of the MLP-Mixer (Tolstikhin et al., 2021) and the Vision Transformer (Dosovitskiy et al., 2020) to MRI (Mansour et al., 2022; Lin & Heckel, 2022) ablate their performance as a function of the training set size and find that non-CNN based approaches require more data to achieve a similar performance, but potentially also benefit more from even larger datasets.

Original: However, those experiments are run with fixed model sizes and only 3-4 realizations of training set size thus it is unclear when performance saturates.
Revised: However, those experiments are run with fixed model sizes and only 3-4 realizations of training set size thus it is unclear when performance saturates.

Original:  E MPIRICAL SCALING LAWS FOR DENOISING
Revised:  E MPIRICAL SCALING LAWS FOR DENOISING

Original: In this section, we consider the problem of estimating an image x from a noisy observation y =
Revised: In this section, we consider the problem of estimating an image x from a noisy observation y =

Original: x + z , where z is additive Gaussian noise z ∼ N (0 , σ 2 z I ) .
Revised: x + z , where z is additive Gaussian noise z ∼ N (0 , σ 2 z I ) .

Original: Neural networks trained end-to-end to map a noisy observation to a clean image perform best, and outperform classical approaches likethe self-similarity driven BM3D (Dabov et al., 2007) and non-local means (Buades et al., 2005) that are not based on training data.
Revised: Neural networks trained end-to-end to map a noisy observation to a clean image perform best, and outperform classical approaches like BM3D (Dabov et al., 2007) and non-local means (Buades et al., 2005) that are not based on training data.

Original: Datasets.
Revised: Datasets.

Original: To be able to study the performance of a learned denoiser over a wide range of training set sizes we work with the ImageNet dataset (Russakovsky et al., 2015).
Revised: To enable studying learned denoisers over a wide range of training set sizes we work with the ImageNet dataset (Russakovsky et al., 2015).

Original: The ImageNet training set contains 1,281,167 images of 1000 different object classes.
Revised: Its training set contains 1.3M images ofdifferent classes.

Original: We reserve 20 randomly selected classes for validation and testing.
Revised: We reserve 20 random classes for validation and testing.

Original: From the images of the remaining classes we design 10 training set subsets S N of sizes N ∈ [100 , 100000] (see Fig. 1(a)) and with S i ⊆
Revised: We design 10 training set subsets S N of sizes N ∈ [100 , 100000] (see Fig. 1(a)) and with S i ⊆

Original: S j for i ≤ j .
Revised: S j for i ≤ j .

Original: To make the distributions of images between the subsets as homogeneous as possible, the number of images from different classes within a subset differ by at most one.
Revised: To make the distributions of images between the subsets as homogeneous as possible, the number of images from different classes within a subset differ by at most one.

Original: The pixel intensities range from 0 to 255 and we generate noisy images by adding Gaussian noise with variance of σ z = 25 , which results in noisy images with 20.17dB  PSNR.
Revised: We generate noisy images by adding Gaussian noise with variance of σ z = 25 (pixels in the range from 0 to 255), i.e. 20.17dB in PSNR.

Original: Model variants and training.
Revised: Model variants and training.

Original: We train a CNN-based U-Net (Ronneberger et al., 2015), a detailed description of the model architecture is in Appx.
Revised: We train a CNN-based U-Net (Ronneberger et al., 2015), a detailed description of the model architecture is in Appx.

Original: A.1.
Revised: A.1.

Original: We also train the Swin-Transformer (Liu et al., 2021) based SwinIR model
Revised: We also train the Swin-Transformer (Liu et al., 2021) based SwinIR model

Original: (Liang et al., 2021) that achieves SOTA for a variety of image reconstruction tasks, including denoising.
Revised: (Liang et al., 2021) that achieves SOTA for a variety of image reconstruction tasks, including denoising.

Original: This model is interesting since transformers scale well with the number of training examples for other domains, e.g., for image classification pre-training (Dosovitskiy et al., 2020).
Revised: This model is interesting since transformers scale well with the number of training examples for other domains, e.g., for image classification pre-training (Dosovitskiy et al., 2020).

Original: For U-Net and SwinIR we vary the number of network parameters as depicted in Fig.
Revised: For U-Net and SwinIR we vary the number of network parameters as depicted in Fig.

Original: 1 (b) and Fig.(b) respectively.
Revised: 1 (b) and Fig.(b) respectively.

Original: Appx.
Revised: Appx.

Original: A.1 and A.3 contain detailed descriptions on how the models are trained and the network size is adapted.
Revised: A.1 and A.3 contain detailed descriptions on how the models are trained and the network size is adapted.

Original: Results and discussion.
Revised: Results and discussion.

Original: For each training set size Fig. 1(a) and Fig. 2(a) show the reconstruction performancesin PSNR of the best U-Net and SwinIR respectively over all considered network sizes.
Revised: Fig. 1(a) and Fig. 2(a) show the reconstruction performances of the best U-Net and SwinIR respectively over all considered network sizes.

Original: Our main findings are as follows.
Revised: Our main findings are as follows.

Original: A training set size of around 100 is already sufficient to train a decent image denoiser.
Revised: A training set size of around 100 is already sufficient to train a decent image denoiser.

Original: Beyond 100, we can fit a linear power law to the performance of the U-Net with a scaling coefficient α = 

Revised: Beyond 100, we can fit a linear power law to the performance of the U-Net with a scaling coefficient α =

Original: .that  holds up to training set sizes of about 6k images.
Revised: .that approximately holds up to training set sizes of about 6k images.

Original: For training set sizes larger than 6k we can fit a second linear power lawbut with significantly smaller scaling coefficient α = 0 .
Revised: Beyond 6k, we can fit a second linear power law with significantly smaller scaling coefficient α = 0 .

Original: Similarly, for the SwinIR we can fit a power law with coefficient α = 0 .
Revised: Similarly, for the SwinIR we can fit a power law with coefficient α = 0 .

Original: 0050 in the regime of little training data and a power law with significantly smaller coefficient α = 0 .
Revised: 0050 in the regime of little training data and a power law with significantly smaller coefficient α = 0 .

Original: 0014 in the regime of moderate training data, thus the two architectures scale essentially equivalently.
Revised: in the regime of moderate training data, thus the two architectures scale essentially equivalently.

Original: We conclude that while denoising benefits from more training examples, the drop in scaling coefficient indicates that scaling from thousands tohundreds of millions of imagesas they are currently used for image classification is only expected to yield a marginal gain in performance.
Revised: While denoising benefits from more training examples, the drop in scaling coefficient indicates that scaling from thousands to millions of images is only expected to yield relatively marginal performance gains.

Original: While the gains are small and we expect gains from further scaling to be even smaller, our largest SwinIR trained on 100k images yields a new SOTA for Gaussian color image denoising.
Revised: While the gains are small and we expect gains from further scaling to be even smaller, our largest SwinIR trained on 100k images yields a new SOTA for Gaussian color image denoising.

Original: On 4 common test sets it achieves an improvement between 0.05 to 0.21dB, see Appx.
Revised: On four common test sets it achieves an improvement between 0.05 to 0.22dB, see Appx.

Original: B. 
Revised: B. Visualizations of reconstructions from models along the curves in Fig. 1(a) and Fig. 2(a) can be found in Fig. 4, Appx. A.

Original: From the comparison in Fig. 2(a), we further conclude that the effect of large training sets on the performance of the U-Net can not make up for the improved modeling error that stems from the superior network architecture of the SwinIR.
Revised: Comparing the U-Net scaling to SwinIR scaling (Fig. 2(a)) we see that the effect of large training sets on the performance of the U-Net can not make up for the improved modeling error that stems from the superior network architecture of the SwinIR.

Original: In fact, the simulated scaling coefficient α = 0 .predicts
Revised: In fact, the simulated scaling coefficient α = 0 .predicts

Original: a required training set size of about 100M images for U-Net to achieve the best performance of the SwinIR, and that is an optimistic prediction since it assumes that the scaling does not slow down further for another 3 orders of magnitude.
Revised: a required training set size of about 150M images for U-Net to achieve the best performance of the SwinIR, and that is an optimistic prediction since it assumes that the scaling does not slow down further for another 3 orders of magnitude.

Original: Thus,even relatively modest model improvements such as those obtained by transformers over convolutional networks are larger than what we expect from scaling the number of training examples from tens of thousands to millions.
Revised: Thus, model improvements such as those obtained by transformers over convolutional networks are larger than what we expect from scaling the number of training examples from tens of thousands to millions.

Revised: An interesting question is whether different noise levels lead to different data requirements, as indicated by a different scaling behavior.

Revised: We investigate the performance of U-Net for a smaller noise variance σ z = 15 in Appx.

Revised: D.2.

Revised: While the results are still preliminary, the qualitative scaling behavior is similar; the difference are that the scaling law pertaining to smaller noise is slightly steeper, and overall the performance improves by about 1.3dB in PSNR.

Original: Finally, note that for our results we choose to re-sample the noise per training image in every training epoch, since it makes the best use of the available clean images.
Revised: Finally, note that for our results we choose to re-sample the noise per training image in every training epoch, since it makes the best use of the available clean images.

Original: However, fixing the noise is also interesting, since it is more similar to a setup in which the noise statistics are unknown like in real-world noise removal.
Revised: However, fixing the noise is also interesting, since it is more similar to a setup in which the noise statistics are unknown like in real-world noise removal.

Original: We refer to Appx.
Revised: See Appx.

Original: D fora detailed discussion of additional results, where the noise is fixed.
Revised: D.1 for additional results where the noise is fixed.

Original: We found that compared to re-sampling the noise the performance of a U-Net drops by 0.3dB for small training set sizes.
Revised: We found that compared to re-sampling the noise the performance of a U-Net drops by 0.3dB for small training set sizes.

Original: As the training set size increases the performance approaches the one of re-sampling the noise resulting in a steeper scaling law that flattens at slightly larger training set sizes.
Revised: As the training set size increases the performance approaches the one of re-sampling the noise resulting in a steeper scaling law that flattens at slightly larger training set sizes.

Original: Robustness of our finding of slowing scaling laws.
Revised: Robustness of our finding of slowing scaling laws.

Original: Our main finding for denoising is that the scaling laws slow at relatively small amounts of training data points (i.e., the scaling coefficient becomes very small).
Revised: Our main finding for denoising is that the scaling laws slow at relatively small amounts of training data points (i.e., the scaling coefficient becomes very small).

Original: We next argue why we think that this is a robust finding.
Revised: We next argue why we think that this is a robust finding.

Original: In principle it could be that the networks are not sufficiently large for the performance to improve further as the dataset size increases.
Revised: In principle it could be that the networks are not sufficiently large for the performance to improve further as the dataset size increases.

Original: However, for the U-Net and training set sizes of 10k and 30k the performance increase already slows, and for those training set sizes, increasing the network size does not improve performance, as shown in Fig. 1(b).
Revised: However, for the U-Net and training set sizes of 10k and 30k the performance increase already slows, and for those training set sizes, increasing the network size does not improve performance, as shown in Fig. 1(b).

Original: For smaller network sizes the curves in Fig.
Revised: For smaller network sizes the curves in Fig.

Original: 1(b) first increase and then decrease indicating that we found network sizes close to the optimum.
Revised: 1(b) first increase and then decrease indicating that we found network sizes close to the optimum.

Original: The parameter scaling of the SwinIR in Fig. 2(b) indicates that for training set sizes of 10k and 100k the performance still benefits from larger networks.
Revised: The parameter scaling of the SwinIR in Fig. 2(b) indicates that for training set sizes of 10k and 100k the performance still benefits from larger networks.

Original: Hence, the power law for those training set sizes in Fig.
Revised: Hence, the power law for those training set sizes in Fig.

Original: 2(a) can become slightly steeper once we further increase the network size.
Revised: 2(a) can become slightly steeper once we further increase the network size.

Original: However, for the slope of the flat power law in (a) to reach the slope of the steep power law, the parameter scaling in (b) for 100k training images would need to hold for another 3 orders of magnitude, i.e. about 90B instead of the currently used 129M network parameters, which is very unlikely considering that the current network sizes already suffice to saturate the performance for small/moderate training set sizes.
Revised: However, for the slope of the flat power law in (a) to reach the slope of the steep power law, the parameter scaling in (b) for 100k training images would need to hold for another 2 orders of magnitude, i.e. about 12B instead of the currently used 129M network parameters, which is very unlikely considering that the current network sizes already suffice to saturate the performance for small/moderate training set sizes.

Original: Finally, it could be that with higher quality or higher diversity of the data, the denoising performance would further improve.
Revised: Finally, it could be that with higher quality or higher diversity of the data, the denoising performance would further improve.

Original: In our experiments, we increase the training set sizes by adding more images from the same classes of ImageNet, and we continue to test on different classes.
Revised: In our experiments, we increase the training set sizes by adding more images from the same classes of ImageNet, and we continue to test on different classes.

Original: Thus, it could be that adding training examples from a different, more diverse data source leads to less of a slowing of the scaling law in Fig. 1(a).
Revised: Thus, it could be that adding training examples from a different, more diverse data source leads to less of a slowing of the scaling law in Fig. 1(a).

Original: We test this hypothesis by scaling our training set with 3k images to 10k images not by adding more images from ImageNet but images from the datasets that are used to train the original SwinIR, i.e., we add all images from DIV2K (Agustsson & Timofte, 2017),
Revised: We test this hypothesis by scaling our training set with 3k images to 10k images not by adding more images from ImageNet but images from the datasets that are used to train the original SwinIR, i.e., we add all images from DIV2K (Agustsson & Timofte, 2017),

Original: Flickr2K (Timofte et al., 2017) and BSD500 (Arbel ´ aez et al., 2011) and add the remaining images from WED (Ma et al., 2017).
Revised: Flickr2K (Timofte et al., 2017) and BSD500 (Arbel´aez et al., 2011) and add the remaining images from WED (Ma et al., 2017).

Original: Keeping all other hyperparameters fixed the U-Net obtains for this dataset a PSNR of 32.239dB, which is slightly worse than the 32.252dB it achieves with our training set with 10k images only from ImageNet.
Revised: Keeping all other hyperparameters fixed the U-Net obtains for this dataset a PSNR of 32.239dB, which is slightly worse than the 32.252dB it achieves with our training set with 10k images only from ImageNet.

Original: Hence, we reject the hypothesis that the drop in scaling coefficients can be explained with how the training sets are designed in our experiment.
Revised: Hence, we reject the hypothesis that the drop in scaling coefficients can be explained with how the training sets are designed in our experiment.

Original:  E MPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING
Revised:  E MPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING

Original: We consider compressive sensing  to achieve 4x accelerated MRI.
Revised: We consider compressive sensing (CS) to achieve 4x accelerated MRI.

Original: MRI is an important medical imaging technique for its non-invasiveness and high accuracy.
Revised: MRI is an important medical imaging technique for its non-invasiveness and high accuracy.

Original: Due to the physics of the measurement process, MRI is inherently slow.
Revised: Due to the physics of the measurement process, MRI is inherently slow.

Original: Accelerated MRI aims at significantly reducing the scan time by undersampling the measurements.
Revised: Accelerated MRI aims at significantly reducing the scan time by undersampling the measurements.

Original: In addition, MRI scanners rely on parallel imaging, which uses multiple receiver coils to simultaneously collect different measurements of the same object.
Revised: In addition, MRI scanners rely on parallel imaging, which uses multiple receiver coils to simultaneously collect different measurements of the same object.

Original: This leads to the following reconstruction problem.
Revised: This leads to the following reconstruction problem.

Original: We are given measurements y i ∈
Revised: We are given measurements y i ∈

Original: C m of the image x ∈ C n as y i
Revised: C m of the image x ∈ C n as y i

Original: = MFS i x
Revised: = MFS i x

Original: + noise i for i = 1 , ..., C , and our goal is to reconstruct the image from those measurements.
Revised: + noise i for i = 1 , ..., C , and our goal is to reconstruct the image from those measurements.

Original: Here, C is the number of receiver coils, the diagonal matrices S i ∈ C n × n are the sensitivity maps modelling the signal strength perceived by the i -th coil, F ∈ C n × n is the discrete Fourier transform (DFT), and M ∈ C m × n is a binary mask implementing the undersampling.
Revised: Here, C is the number of receiver coils, the diagonal matrices S i ∈ C n × n are the sensitivity maps modelling the signal strength perceived by the i -th coil, F ∈ C n × n is the discrete Fourier transform (DFT), and M ∈ C m × n is a binary mask implementing the undersampling.

Original: Classical compressed sensing approaches (Lustig et al., 2008) first estimate the sensitivity maps with a method such as ESPIRiT (Uecker et al., 2014), and second estimate the unknown image by solving a regularized optimization problem, such as total-variation norm minimization.
Revised: Classical CS approaches (Lustig et al., 2008) first estimate the sensitivity maps with a method such as ESPIRiT (Uecker et al., 2014), and second estimate the unknown image by solving a regularized optimization problem, such as total-variation norm minimization.

Original: Recently, deep learning based methods have been shown to significantly outperform classical compressed sensing methods due to their ability to learn more complex and accurate signal models from data. The 2019 and 2020 editions of the fastMRI challenge document the success of learning based methods for accelerated MRI (Knoll et al., 2020a; Muckley et al., 2021).
Revised: Recently, deep learning based methods have been shown to significantly outperform classical CS methods due to their ability to learn more complex and accurate signal models from data (Knoll et al., 2020a; Muckley et al., 2021).

Original: Datasets.
Revised: Datasets.

Original: To explore the performance of learning based MR reconstruction over a wide range of training set sizes, we use the fastMRI multi-coil brain dataset (Knoll et al., 2020b), which is the largest publicly available dataset for MRI.
Revised: To explore the performance of learning based MR reconstruction over a wide range of training set sizes, we use the fastMRI multi-coil brain dataset (Knoll et al., 2020b), which is the largest publicly available dataset for MRI.

Original: The dataset consists of images of different contrasts.
Revised: The dataset consists of images of different contrasts.

Original: To ensure that the statistics of the dataset are as homogeneous as possible, we take the subset of the dataset corresponding to a single contrast resulting in 50k training images.
Revised: To ensure that the statistics of the dataset are as homogeneous as possible, we take the subset of the dataset corresponding to a single contrast resulting in 50k training images.

Original: We design training set subsets S N of size N ∈ [50 , 50000]
Revised: We design training set subsets S N of size N ∈ [50 , 50000]

Original: (see Fig.
Revised: (see Fig.

Original: 1 (c)) with S i ⊆
Revised: 1 (c)) with S i ⊆

Original: S j for i ≤ j .
Revised: S j for i ≤ j .

Original: For more information on selection, division and subsampling of the dataset see Appx.
Revised: For more information on selection, division and subsampling of the dataset see Appx.

Original: A.2.
Revised: A.2.

Original: Model variants and training.
Revised: Model variants and training.

Original: We train the same U-Net model used in Section 3 and described in Appx.
Revised: We train the same U-Net model used in Section 3 and described in Appx.

Original: A.1 but with 4 blocks per encoder/decoder.
Revised: A.1 but with 4 blocks per encoder/decoder.

Original: The network is trained end-to-end to map a coarse reconstruction ˆ x CR to the ground truth image x .
Revised: The network is trained end-to-end to map a coarse reconstruction ˆ x CR to the ground truth image x .

Original: The coarse reconstruction is obtained as ˆ x CR =
Revised: The coarse reconstruction is obtained as ˆ x CR =

Original: (cid:16)(cid:80)
Revised: (cid:16)(cid:80)

Original: Ci =1 | F − 1 y i, ZF | 2 (cid:17) 1 / 2 , where F − 1 is the inverse DFT, and
Revised: Ci =1 | F − 1 y i, ZF | 2 (cid:17) 1 / 2 , where F − 1 is the inverse DFT, and

Original: y i, ZF are the undersampled measurements of the i -th coil, where missing entries are filled with zeros.
Revised: y i, ZF are the undersampled measurements of the i -th coil, where missing entries are filled with zeros.

Original: We vary the number of network parameters as depicted in Fig.
Revised: We vary the number of network parameters as depicted in Fig.

Original: 1 (d).
Revised: 1 (d).

Original: Appx.
Revised: Appx.

Original: A.2 contains detailed descriptions on how the model is trained and the network size is adapted.
Revised: A.2 contains detailed descriptions on how the model is trained and the network size is adapted.

Original: Results and discussion.
Revised: Results and discussion.

Original: For each training set size Fig.
Revised: For each training set size Fig.

Original: 1 (c) shows the reconstruction performance in structural similarity (SSIM) of the best model over all simulated network sizes.
Revised: 1 (c) shows the reconstruction performance in structural similarity (SSIM) of the best model over all simulated network sizes.

Original: There are 2 main findings.
Revised: There are 2 main findings.

Original: First, we can fit a linear power law with a scaling coefficient α = 0 .
Revised: First, we can fit a linear power law with a scaling coefficient α = 0 .

Original: 0072 that holds up to training set sizes of about 2.5k images.
Revised: 0072 that holds up to training set sizes of about 2.5k images.

Original: Second, for training set sizes starting from 5k we can fit a second linear power law but with significantly smaller scaling coefficient α = 0 .
Revised: Second, for training set sizes starting from 5k we can fit a second linear power law but with significantly smaller scaling coefficient α = 0 .

Original: Similar to denoising in Section 3 we conclude that while accelerated MRI slightly benefits form more training examples, the drop in scaling coefficient indicates that it is unlikely that even training set sizes of the order of hundreds of millions of images result in substantial gains. 
Revised: Similar to denoising in Section 3 we conclude that while accelerated MRI slightly benefits form more training examples, the drop in scaling coefficient indicates that it is unlikely that even training set sizes of the order of hundreds of millions of images result in substantial gains. Visualizations of reconstructions from models along the curve in Fig. 1 (c) can be found in Fig. 5, Appx.

Revised: A.2.

Original: Robustness of our finding of slowing scaling laws.
Revised: Robustness of our finding of slowing scaling laws.

Original: Next, we argue why the drop in scaling coefficient for accelerated MRI is a robust finding.
Revised: Next, we argue why the drop in scaling coefficient for accelerated MRI is a robust finding.

Original: In Fig.
Revised: In Fig.

Original: 1 (d) we demonstrate that the  sizeof the investigated networks does not bottleneck the performance of our largest training sets.
Revised: 1 (d) we demonstrate that the network size does not bottleneck the performance of our largest training sets.

Original: For each training set size the performance as a function of the number of network parameters is relatively flat.
Revised: For each training set size the performance as a function of the number of network parameters is relatively flat.

Original: Even small training sets benefit from large networks before their performance slightly decays for very large networks.
Revised: Even small training sets benefit from large networks before their performance slightly decays for very large networks.

Original: Hence, we expect that for large training sets the performance as a function of network parameters would not increase significantly before decaying.
Revised: Hence, we expect that for large training sets the performance as a function of network parameters would not increase significantly before decaying.

Original: Finally, is there a different type of training data that would improve the scaling coefficient?
Revised: Finally, is there a different type of training data that would improve the scaling coefficient?

Original: We don’t think so, since all experiments are for the very specific task of reconstructing brain images of one particular contrast, which means that the examples that we add to increase the size of the training sets is already the data most suitable to solve this task.
Revised: We don’t think so, since all experiments are for the very specific task of reconstructing brain images of one particular contrast, which means that the examples that we add to increase the size of the training sets is already the data most suitable to solve this task.

Original:  U NDERSTANDING SCALING LAWS FOR DENOISING THEORETICALLY
Revised:  U NDERSTANDING SCALING LAWS FOR DENOISING THEORETICALLY

Original: Wenext study a simple linear denoisertheoretically that is trained end-to-end to reconstruct a clean image from a noisy observation,  as a function of the number of training examples.
Revised: We study a simple linear denoiser that is trained end-to-end to reconstruct a clean image from a noisy observation, in order to understand how the error as a function of the number of training examples for inverse problems is expected to look like.

Original: We define a joint distribution over a signal and the corresponding measurement ( x , y ) as follows.
Revised: We define a joint distribution over a signal and the corresponding measurement ( x , y ) as follows.

Original: Consider a d < n dimensional subspace of R n parameterized by the orthonormal basis U ∈ R n × d .
Revised: Consider a d < n dimensional subspace of R n parameterized by the orthonormal basis U ∈ R n × d .

Original: First, we draw a signal approximately uniformly from the subspace x =
Revised: We draw a signal approximately uniformly from the subspace x =

Original: Uc , where c ∼ N (0 , I ) . Second, we generate corresponding noisy measurement as y = x
Revised: Uc , where c ∼ N (0 , I ) , and an associated noisy measurement as y = x

Original: + z , where z ∼ N (0 , σ 2 z I ) is Gaussianmeasurement noise.
Revised: + z , where z ∼ N (0 , σ 2 z I ) is Gaussian noise.

Original: We assume that we are given a training set { ( x 1 , y 1 ) , . . .
Revised: The subspace is unknown, but we are given a training set { ( x 1 , y 1 ) , . . .

Original: , ( x N , y N ) } , consisting oftraining examples drawn iid from the joint distribution over ( x , y ) .
Revised: , ( x N , y N ) } , consisting of examples drawn iid from the joint distribution over ( x , y ) .

Original: Our signal model assumes that the signal lies in a low dimensional subspace.
Revised: We assume that the signal lies in a low dimensional subspace.

Original: Assuming that data lies in a low-dimensional subspace or more general a union of low-dimensional  isrelatively common, e.g., it underlies the denoising of natural images via wavelet thresholding (Donoho & Johnstone, 1995; Simoncelli & Adelson, 1996;
Revised: Assuming that data lies in a lowdimensional subspace or more general, a union of low-dimensional subspaces, is common, for example it underlies the denoising of natural images via wavelet thresholding (Donoho & Johnstone, 1995; Simoncelli & Adelson, 1996;

Original: Chang et al., 2000).
Revised: Chang et al., 2000).

Original: Mohan et al.
Revised: Mohan et al.

Original: (2020) found that even deep learning based denoisers implicitly perform a projection onto an adaptively-selected low-dimensional subspace that captures the features of natural images.
Revised: (2020) found that even deep learning based denoisers implicitly perform a projection onto an adaptively-selected low-dimensional subspace that captures the features of natural images.

Original: We consider a linear estimator of the form f W ( y ) =
Revised: We consider a linear estimator of the form f W ( y ) =

Original: Wy , and  performanceis measured in terms of the expected mean-squared reconstruction error:
Revised: Wy , and measure performance in terms of the expected mean-squared reconstruction error, defined as R ( W ) =

Original: [Equation] normalized by the latent signal dimension d , where expectation is over the joint distribution of ( x , y ) . We aim at learning the weights W end-to-end by applying gradient descent to the loss
Revised: 1 d E (cid:104) ∥

Original: [Equation] [Figure] where X , Y ∈ R n × N contain the training examples as columns.
Revised: Wy − x ∥ 22 (cid:105) , where expectation is over the joint distribution of ( x , y ) .

Original: The optimal linear estimator.
Revised: The optimal linear estimator.

Original: Before analyzing the estimator learned through gradient descent, we note that the optimal linear estimator (i.e., the estimator that minimizes the risk defined in equation (1)) is given by W ∗ = 1 1+ σ 2 z UU T .
Revised: The optimal linear estimator (i.e., the estimator that minimizes the risk R ) is given by W ∗ = 1 1+ σ 2 z UU T .

Original: The estimator projects the data onto the subspace and shrinks towards zero, depending on the noise variance.
Revised: The estimator projects the data onto the subspace and shrinks towards zero, depending on the noise variance.

Original: The  riskof that estimator is R ( W ∗ ) =
Revised: The associated risk is R ( W ∗ ) =

Original: σ 2 z / (1 + σ 2 z ) .
Revised: σ 2 z / (1 + σ 2 z ) .

Original: An estimator based on subspace estimation.
Revised: An estimator based on subspace estimation.

Original: The optimal estimator W ∗ requires knowledge of the unknown subspace.
Revised: The optimal estimator W ∗ requires knowledge of the unknown subspace.

Original: While the subspace is not known, we could learn it from the noisy data as follows. We can perform principal component analysis on Y =
Revised: While the subspace is not known, we can learn it from the noisy data by performing principal component analysis on Y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y N ] , i.e., estimate the subspace as the d leading singular vectors of the empirical co-variance matrix YY T ∈ R n × n . The corresponding estimate is denoted by  ˆ U  .
Revised: , y N ] . Specifically, we estimate the subspace as the d -leading singular vectors of the empirical co-variance matrix YY T ∈ R n × n , denoted by by ˆ U ∈ R n × d .

Original: If the measurement noise is zero (i.e., σ 2 z = 0 ), this algorithm perfectly reconstruct an orthonormal basis for the d -dimensional subspace  provided we observe at least d many linearly independent signals, which occurs with probability one if we draw data according to the model defined above, and if the number of training examples obeys N ≥ d .
Revised: If the measurement noise is zero (i.e., σ 2 z = 0 ) , ˆ U is an orthonormal basis for the d -dimensional subspace U provided we observe at least d many linearly independent signals, which occurs with probability one if we draw data according to the model defined above, and if the number of training examples obeys N ≥ d .

Revised: There is a vast literature on PCA; see Vershynin (2011; 2018); Rudelson & Vershynin (2010); Tropp (2012) for tools from high-dimensional probability to analyze the PCA estimate.

Original: Consider now the estimator W PCA = {+1 1+ σ 
+}
Revised: Now consider the estimator W PCA = {+1 1+ σ 
+}

Original: [Figure] + σ 2 z
Revised: z

Original: ˆ U
Revised: ˆ U

Original: ˆ U T y based on N noisy training points.
Revised: ˆ U T y based on N noisy training points.

Original: The estimator assumes knowledge of the noise variance, but it is not difficult to estimate it relatively accurately.
Revised: The estimator assumes knowledge of the noise variance, but it is not difficult to estimate it relatively accurately.

Original: The following result, proven in Appx.
Revised: The following result, proven in Appx.

Original: F, characterizes therisk associated with the estimator.
Revised: F, characterizes the associated risk.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that the number of training examples obeys ( d + nσ 2 z ) log( n ) ≤ N .
Revised: Suppose that the number of training examples obeys ( d + nσ 2 z ) log( n ) ≤ N .

Original: For a numerical constant c , with probability at least 1 − n − 10 − 3 e − d + e − n , the risk of the PCA-estimate is bounded by
Revised: For a numerical constant c , with probability at least 1 − n − 10 − 3 e − d + e − n , the risk of the PCA-estimate is bounded by

Original: [Equation]
Revised: [Equation]

Original: Thus, as long as the number of training examples N is sufficiently large relative to ( d + nσ 2 z ) , the risk of this estimator is close to the risk of the optimal estimator.
Revised: Thus, as long as the number of training examples N is sufficiently large relative to ( d + nσ 2 z ) , the risk of the PCA-estimator is close to the risk of the optimal estimator.

Original: Estimator learned end-to-end.
Revised: Estimator learned end-to-end.

Original: We now consider the estimator that applies gradient descent to the loss (2), and is regularized via early stopping the gradient descent iterations.
Revised: We now consider an estimator learned end-to-end, by applying gradient descent to the empirical risk L ( W ) = (cid:80) Ni = 1 ∥ Wy i − x i ∥ 22 and we regularize via earlystopping the gradient descent iterations.

Original:  Theorem 2.
Revised: The risk of the estimate after k iterations of gradient descent, W k , is bounded by the next result. This estimator mimics the supervised training we consider throughout this section, with the difference that here we consider a simple neural network, and in the previous sections we trained a neural network. Theorem 2.

Original: Let W k be the matrix obtained by applying k iterations of gradient descent with stepsize η starting at W 0 = 0 to the loss L ( W ) .
Revised: Let W k be the matrix obtained by applying k iterations of gradient descent with stepsize η starting at W 0 = 0 to the loss L ( W ) .

Original: Suppose the number of training examples obeys ( d + nσ 2 z ) log( n ) ≤ N  and N log( N ) ≤ n .
Revised: Consider the regime where the number of training examples obeys ( d + nσ 2 z ) log( n ) ≤ N ≤ ξd/σ 2 z for an arbitrary ξ and N log( N ) ≤ n .

Original: For an appropriate choice of the stepsize η there exists an optimal early stopping time k opt at whichwe upper-bound the risk of the estimator f W ( y )= W k opt y with probability at least 1 − 2 e [Equation] N/
Revised: For an appropriate choice of the stepsize η there exists an optimal early stopping time k opt at which the risk of the estimator f W ( y )

Original: [Equation] N [Equation] n
Revised: =

Original: [Equation] e − d [Equation] n − e − N − 2 e − n/ 2 as
Revised: W k opt y is upper-bounded with probability at least 1 − 2 e − N/ 8 − 2 e − N/ 18 − 5 n − 9 − 5 e − d − 2 e − n − e − N − 2 e − n/ 2 by

Original: [Equation]
Revised: [Equation]

Original: , where c is a numerical constant.
Revised: where c is a numerical constant.

Original: [Figure]

Original: The proof is provided in Appx.
Revised: The proof is provided in Appx.

Original: G. Similar to Thm.
Revised: G. Similar to Thm.

Original: 1 the first term in the bound corresponds to the noise-dependent error floor and the second term to a factor d/N that represents the error induced by learning the signal model with a finite number of training examples.
Revised: 1 the first term in the bound corresponds to the noise-dependent error floor and the second two terms decrease in ( d + nσ 2 z ) /N and represent the error induced by learning the estimator with a finite number of training examples.

Original: Once this error is small relative to the error floor, more training examples do not improve performance.
Revised: Once this error is small relative to the error floor, more training examples do not improve performance.

Original: The last term in the bound corresponds to the error induced by the noise that is already fitted at early stopping iteration k opt .
Revised: See Appx. E for a more detailed discussion on the estimator and the assumptions of Thm. 2. Discussion.

Original:  Fig.
Revised: In Fig.

Original: 3 numerically demonstrates the regularizing effect of early stopping gradient descent, where
Revised: 3, we plot the risk as a function of the training set size for the early-stopped empirical risk minimization (ERM) and the PCA based estimator.

Original: W ∞ = XY † is the converged learned estimator (see Appx.
Revised: Starting at a training set size of N = 100 , we see that the risk minus the optimal risk follows approximately a power law, i.e., log( R ( W )

Original: G, Eq, (4)).
Revised: − R ( W ∗ ))

Original: For the optimally stopped estimator W k opt we see three regions, coined in Hestness et al.
Revised: ≈ − α log( N ) , as suggested by Thms. 1 and 2. In practice, however, we don’t know the risk of the optimal estimator, R ( W ∗ ) .

Original: The training set size does not suffice to learn a well-performing mapping ( small data region ). The performance decays as  N  αwith α < 0 ( power law region ). Finally, the performance converges to the problem specific irreducible error R ( W ∗ ) ( irreducible error region ).
Revised: Therefore, in the second row of Fig. 3 and throughout our empirical results we plot the risk log( R ( W )) as a function of the number of training examples log( N ) and distinguish different regions by fitting approximated scaling coefficients α to log( R ( W )) ≈ − α log( N ) .

Original: The numerical simulations in Fig.
Revised: In our theoretical example, we can identify three regions, coined in Hestness et al.

Original: 4 indicate that the early stopped learned estimator W k opt outperforms the subspace estimator W PCA in  the regime, where both estimators have not converged to  the  irreducible error yet.
Revised: The small data region in which the training set size does not suffice to learn a well-performing mapping and the power-law region in which the performance decays approximately as N α with α < 0 , and an problem-specific irreducible error region, which is

Original: The scaling coefficients of W k opt consistently exceed the coefficients of W PCA .
Revised: R ( W ∗ ) here. Note that the power-law coefficient in the risk plot R ( W ) are smaller then those when plotting R ( W ) − R ( W ∗ ) , an are only approximate scaling coefficients.

Original: Already in this highly simplified setup of an inverse problem the true scaling coefficients heavily depend on model parameters such as the noise variance as can be seen in Fig.
Revised: Already in this highly simplified setup of an inverse problem the true scaling coefficients heavily depend on model parameters such as the noise variance as can be seen in Fig.

Original: The scaling coefficients further vary with the signal dimension d and ambient dimension n (see Appx.
Revised: The scaling coefficients further vary with the signal dimension d and ambient dimension n (see Appx.

Original: E).
Revised: E.1).

Original: D ISCUSSION AND OUTLOOK
Revised: [Figure]  C ONCLUSION

Original: We found that the performance improvement of deep learning based image reconstruction as a function of the number of training examples slows already at moderate training set sizes, indicating that only marginal gains are expected beyond a few thousand examples. 
Revised: We found that the performance improvement of deep learning based image reconstruction as a function of the number of training examples slows already at moderate training set sizes, indicating that only marginal gains are expected beyond a few thousand examples. Limitations. This finding is based on studying three different reconstruction problems (denoising, super-resolution [Figure]

Revised: and compressive sensing), two architectures (U-net and SwinIR)

Original: Our findings pertain to standard in-distribution evaluation, i.e., the test images are from the same distribution as the training images.
Revised: [Figure] and for each setup we extensively optimized hyperparameters and carefully scaled the networks. Scaling gave new state-of-the-art results on four denoising datasets, which provides some confidence in the setup. Moreover, our statements necessarily pertain to the architectures and metrics we study.

Revised: It is possible that other architectures or another scaling of architectures can yield larger improvements when scaling architectures.

Revised: It is also widely acknowledged that image quality metrics (such as SSIM and PSNR) do not fully capture the perceived image quality by humans, and it could be that more training data yields improvements in image quality that are not captured well by SSIM and PSNR.

Revised: Most importantly, our findings pertain to standard in-distribution evaluation, i.e., the test images are from the same distribution as the training images.

Original: To achieve robustness against testing on images out of the training distribution recent work indicates a positive effect of larger and more diverse training sets (Miller et al., 2021; Darestani et al., 2022; Nguyen et al., 2022).
Revised: To achieve robustness against testing on images out of the training distribution recent work indicates a positive effect of larger and more diverse training sets (Miller et al., 2021; Darestani et al., 2022; Nguyen et al., 2022).

Original: Thus it is possible that the out-of-distribution performance of image reconstruction methods improves when trained on a larger and a more diverse dataset, even though the in-distribution performance does not improve further. 
Revised: Thus it is possible that the out-of-distribution performance of image reconstruction methods improves when trained on a larger and a more diverse dataset, even though the in-distribution performance does not improve further. Future research.

Original: Finally, we focus on supervised image reconstruction methods.
Revised: We focus on supervised image reconstruction methods.

Original: For image denoising and other image reconstruction problems self-supervised approaches are also performing very well (Laine et al., 2019; Wang et al., 2022; Zhou et al., 2022), even though supervised methods perform best if clean images are available.
Revised: For image denoising and other image reconstruction problems self-supervised approaches are also performing very well (Laine et al., 2019; Wang et al., 2022; Zhou et al., 2022), even though supervised methods perform best if clean images are available.

Original: It would be very interesting to investigate the scaling laws for self-supervised methods; perhaps more training examples are required to achieve the performance of a supervised setup.
Revised: It would be very interesting to investigate the scaling laws for selfsupervised methods; perhaps more training examples are required to achieve the performance of a supervised setup.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: A D ETAILS OF THE EXPERIMENTAL SETUPS A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISING WITH A U-N ET
Revised: A D ETAILS OF THE EXPERIMENTAL SETUPS A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISING WITH A U-N ET

Revised: In this Section, we give a detailed description of the experimental setup that led to our results for

Revised: Gaussian denoising with a U-Net presented in Fig.1 (a),(b) and Section 3.

Revised: In addition, Fig. 4 shows examples of reconstructions from different models along the performance curve in Fig.

Revised: 1 (a).

Revised: In these examples the improvement in perceived image quality from increasing the training set size from 100 to 1000 is larger than from increasing from 1000 to 10000 or from 10000 to 100000.

Original:  We train U-Nets with two blocks in the encoder and decoder part respectively and skip connections between blocks.
Revised: This correlates with our quantitative findings in Fig. 1 (a). Next, we describe the experimental details. We train U-Nets with two blocks in the encoder and decoder part respectively and skip connections between blocks.

Original: Each block consists of two convolutional layers with LeakyReLU activation and instance normalization (Ulyanov et al., 2017) after every layer, where the number of channels is doubled (halved) after every block in the encoder (decoder).
Revised: Each block consists of two convolutional layers with LeakyReLU activation and instance normalization (Ulyanov et al., 2017) after every layer, where the number of channels is doubled (halved) after every block in the encoder (decoder).

Original: The downsampling in the encoder is implemented as average pooling and the upsampling in the decoder as transposed convolutions.
Revised: The downsampling in the encoder is implemented as average pooling and the upsampling in the decoder as transposed convolutions.

Original: As proposed by Zhang et al.
Revised: As proposed by Zhang et al.

Original: (2017) we train a residual denoiser that learns to predict y − x instead of directly predicting x , which improves performance.
Revised: (2017) we train a residual denoiser that learns to predict y − x instead of directly predicting x , which improves performance.

Original: We scale the network size by increasing the width of the network by scaling the number of channels per (transposed) convolutional layer.
Revised: We scale the network size by increasing the width of the network by scaling the number of channels per (transposed) convolutional layer.

Original: For denoising we trained U-Nets of 7 different sizes.
Revised: For denoising we trained U-Nets of 7 different sizes.

Original: We vary the number of channels in the first layer in { 16 , 32 , 64 , 128 , 192 , 256 , 320 } , which corresponds to { 0 . 1 , 0 . 5 , 1 . 9 , 7 . 4 , 16 . 7 , 30 . 0 , 46 . 5 } million parameters.
Revised: We vary the number of channels in the first layer in { 16 , 32 , 64 , 128 , 192 , 256 , 320 } , which corresponds to { 0 . 1 , 0 . 5 , 1 . 9 , 7 . 4 , 16 . 7 , 30 . 0 , 46 . 5 } million parameters.

Original: We do not vary the depth, since this would change the dimension of the informational bottleneck in the U-Net and thus change the model family itself.
Revised: We do not vary the depth, since this would change the dimension of the informational bottleneck in the U-Net and thus change the model family itself.

Original: The exact training set sizes we consider are { 0 . 1 , 0 . 3 , 0 .
Revised: The exact training set sizes we consider are { 0 . 1 , 0 . 3 , 0 .

Original: 6 , 1 , 3 , 6 , 10 , 30 , 60 , 100 } thousand images from ImageNet.
Revised: 6 , 1 , 3 , 6 , 10 , 30 , 60 , 100 } thousand images from ImageNet.

Original: We center crop the training images to 256 × 256 pixels.
Revised: We center crop the training images to 256 × 256 pixels.

Revised: We also tried a smaller patch size of 128 × 128 pixels, but larger patches showed to have a better performance in the regime of large training set sizes, which is why the results presented in the main body are for patch size 256 × 256 .

Revised: For a comparison between the scaling behavior of the two different patch sizes see Fig.

Original:  We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.
Revised: Appx. D.3. We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.

Original: For validation and testing we use 80 and 300 images respectively taken from 20 random classes that are not used for training.
Revised: For validation and testing we use 80 and 300 images respectively taken fromrandom classes that are not used for training.

Original: We use mean-squared-error loss and Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 .
Revised: We use mean-squared-error loss and Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: For moderate training set sizes up to 3000 images we find that heuristically adjusting the initial learning with the help of an automated learning rate annealing performs well.
Revised: For moderate training set sizes up to 3000 images we find that heuristically adjusting the initial learning with the help of an automated learning rate annealing performs well.

Original: To this end, we start with a learning rate of 10 − 4 and increase after every epoch by a factor of 2 until the validation loss does not improve for 3 consecutive epochs.
Revised: To this end, we start with a learning rate of 10 − 4 and increase after every epoch by a factor of 2 until the validation loss does not improve for 3 consecutive epochs.

Original: We then load the model checkpoint from the learning rate that was still performing well and continue with that learning rate.
Revised: We then load the model checkpoint from the learning rate that was still performing well and continue with that learning rate.

Original: We observe that this scheme typically picks an initial learning rate reduced by factor 2 for every increase in the number of channels
Revised: We observe that this scheme typically picks an initial learning rate reduced by factor 2 for every increase in the number of channels

Original: C .
Revised: C .

Original: For training set sizes larger than 3000 we directly apply this rule to pick an initial learning rate without annealing as we found this to give slightly better results.
Revised: For training set sizes larger than 3000 we directly apply this rule to pick an initial learning rate without annealing as we found this to give slightly better results.

Original: In particular, for number of channels
Revised: In particular, for number of channels

Original: C = { 128 , 192 , 256 , 320 } we used initial learning rates η = { 0 . 0032 , 0 . 0016 , 0 . 0008 , 0 . 0004 } .
Revised: C = { 128 , 192 , 256 , 320 } we used initial learning rates η = { 0 . 0032 , 0 . 0016 , 0 . 0008 , 0 . 0004 } .

Original: For small training sets up to 1000 images we found that batch size of 1 works best.
Revised: For small training sets up to 1000 images we found that batch size of 1 works best.

Original: For larger training sets we use a batch size of 10 and found that further increasing the batch size does not improve performance.
Revised: For larger training sets we use a batch size of 10 and found that further increasing the batch size does not improve performance.

Original: We do not put a limit on the amount of compute for training.
Revised: We do not put a limit on the amount of compute for training.

Original: We use an automated learning rate decay that reduces the learning rate by 0 .
Revised: We use an automated learning rate decay that reduces the learning rate by 0 .

Original: 5 if the validation PSNR has not improved by at least 0.001 forepochs or 6 epochs for training set sizes starting from 6000 images.
Revised: 5 if the validation PSNR has not improved by at least 0.001 forepochs or 6 epochs for training set sizes starting from 6000 images.

Original: Once the learning rate drops to 10 − 5 we observe near to no gains in validation loss and stop the training after 10 additional epochs.
Revised: Once the learning rate drops to 10 − 5 we observe near to no gains in validation loss and stop the training after 10 additional epochs.

Original: For training set sizes up to 1000 we train 3 random seeds and pick the best run.
Revised: For training set sizes up to 1000 we train 3 random seeds and pick the best run.

Original: As the variance between runs compared to the gain in performance between different training set sizes decreases with increasing training set size we only run one seed for larger training set sizes.
Revised: As the variance between runs compared to the gain in performance between different training set sizes decreases with increasing training set size we only run one seed for larger training set sizes.

Original: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA
Revised: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA

Original: Quadro RTX 6000 GPUs.
Revised: Quadro RTX 6000 GPUs.

Original: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 1800 GPU hours for the experiments in Fig.
Revised: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 1800 GPU hours for the experiments in Fig.

Original: 1(a),(b).
Revised: 1(a),(b).

Original: A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING WITH A U-N ET
Revised: A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR COMPRESSIVE SENSING WITH A U-N ET

Revised: In this Section, we give a detailed description of the experimental setup that led to our results for compressive sensing MRI presented in Fig.1 (c),(d) and Section 4.

Revised: In addition, Fig. 5 shows examples of reconstructions from different models along the performance curve in Fig.

Original: For compressed sensing in  the context of accelerated
Revised: 1 (c). In these examples the improvement in perceived image quality from increasing the training set size from 500 to 2500 is larger than from increasing from 2500 to 10000 or from 10000 to 50000. This correlates with our quantitative findings in Fig. 1 (c).

Revised: The first in row in Fig.

Original:  MRI  we trained
Revised: 5 shows an example in which all models including the one trained on the largest training set fail to recover a fine detail. This is a known problem in accelerated MRI and has been documented in both editions of the fast MRI challenge (Knoll et al., 2020a; Muckley et al., 2021) in which for all methods examples could be found in which fine details have not been recovered. However, the question remains if more training data or better models would help or the details are simply not there since the information is lost due to the large undersampling factors considered in the fastMRI challenges and in this work. Next, we describe the experimental details.

Revised: For compressed sensing in the context of accelerated

Original:  U-Nets of 14 different sizes.
Revised: MRI we trained U-Nets of 14 different sizes.

Original: We vary the number of channels in the first layer in { 16 , 32 , 48 , 64 , 96 , 112 , 128 , 144 , 160 , 176 , 192 , 208 , 224 , 256 } , which corresponds to { 2 , 8 , 18 , 31 , 70 , 95 , 124 , 157 , 193 , 234 , 279 , 327 , 380 , 496 } million network parameters.[Table]
Revised: We vary the number of channels in the first layer in { 16 , 32 , 48 , 64 , 96 , 112 , 128 , 144 , 160 , 176 , 192 , 208 , 224 , 256 } , which corresponds to { 2 , 8 , 18 , 31 , 70 , 95 , 124 , 157 , 193 , 234 , 279 , 327 , 380 , 496 } million network parameters.

Original: The exact training set sizes we consider are { 0 . 05 , 0 .
Revised: The exact training set sizes we consider are { 0 . 05 , 0 .

Original: [Table] } thousand AXT weighted images from fastMRI multi-coil brain dataset (Zbontar et al., 2018), where AXT2 corresponds to all images of one type of contrast.
Revised: 5 , 1 , 2 . 5 , 5 , 10 , 25 , 50 } thousand AXT weighted images from fastMRI multi-coil brain dataset (Zbontar et al., 2018), where AXT2 corresponds to all images of one type of contrast.

Original: We focused on images only from this type to make the statistics of our datasets as homogeneous as possible.
Revised: We focused on images only from this type to make the statistics of our datasets as homogeneous as possible.

Original: We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.
Revised: We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.

Original: We use 4732 and 730 additional images for testing and validation.
Revised: We use 4732 and 730 additional images for testing and validation.

Original: We consider an acceleration factor of 4 meaning that we only measure 25% of the information.
Revised: We consider an acceleration factor of 4 meaning that we only measure 25% of the information.

Original: We obtain the 4 times undersampled measurements by masking the fully sampled measurement y with an equispaced mask with 8% center fractions meaning that in the center of y we take all the measurements and take the remaining ones at equispaced intervals.
Revised: We obtain the 4 times undersampled measurements by masking the fully sampled measurement y with an equispaced mask with 8% center fractions meaning that in the center of y we take all the measurements and take the remaining ones at equispaced intervals.

Original: We use structural similarity (SSIM) loss and RMSprop optimizer with α = 0 .
Revised: We use structural similarity (SSIM) loss and RMSprop optimizer with α = 0 .

Original: 99 as this is the default in the fastMRI repository (Zbontar et al., 2018)
Revised: 99 as this is the default in the fastMRI repository (Zbontar et al., 2018)

Original: and we found no improvement by replacing it with
Revised: and we found no improvement by replacing it with

Original: Adam.
Revised: Adam.

Original: We do not put a limit on the amount of compute invested into training.
Revised: We do not put a limit on the amount of compute invested into training.

Original: We deploy an automated learning rate decay that starts at a learning rate of 10 − 3 and decays by a factor 0.1 if the validation SSIM has not improved by at least 10 − 4 for 5 epochs.
Revised: We deploy an automated learning rate decay that starts at a learning rate of 10 − 3 and decays by a factor 0.1 if the validation SSIM has not improved by at least 10 − 4 for 5 epochs.

Original: Once the learning rate drops to 10 −we stop the training after 10 additional epochs.
Revised: Once the learning rate drops to 10 −we stop the training after 10 additional epochs.

Original: Only for the largest training set sizes 25k,50k we found that an additional drop to 10 − 7 resulted in further performance gains.
Revised: Only for the largest training set sizes 25k,50k we found that an additional drop to 10 − 7 resulted in further performance gains.

Original: We use a batch size of 1.
Revised: We use a batch size of 1.

Original: For training set sizes up to 5k we train three models with random seeds and pick the best.
Revised: For training set sizes up to 5k we train three models with random seeds and pick the best.

Original: For larger training set sizes we only run one seed, since the variance between runs decreased.
Revised: For larger training set sizes we only run one seed, since the variance between runs decreased.

Original: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA
Revised: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA

Original: Quadro RTX 6000 GPUs.
Revised: Quadro RTX 6000 GPUs.

Original: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 4250 GPU hours for the experiments in Fig.
Revised: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 4250 GPU hours for the experiments in Fig.

Original: 1 (c),(d).
Revised: 1 (c),(d).

Original: A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISINGWITH THE S WIN IR
Revised: A. E XPERIMENTAL DETAILS FOR EMPIRICAL SCALING LAWS FOR DENOISING

Revised: S WIN IR

Revised: In this Section, we give a detailed description of the experimental setup that led to our results for

Original: To obtain Fig. 
Revised: Gaussian denoising with a SwinIR presented in Fig.2 (a),(b) and Section 3. In addition, Fig. 4 shows examples of reconstructions from different models along the performance curve in Fig. 2 (a). Despite of the best SwinIR clearly outperforming the best U-Net in terms of PSNR

Revised: Clean/Noisy

Revised: U - N e t ∞ PSNR:

Revised: N =30.

Revised: N =30.

Revised: N =31.

Revised: N =31.

Revised: S w i n

Revised: I R ∞ PSNR: 31.

Revised: U - N e t ∞ PSNR: 28.

Revised: S w i n

Revised: I R ∞

Revised: PSNR: 28.

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: it is difficult for the naked eye to notice large differences in the quality of the reconstructions.

Revised: This indicates that for Gaussian denoising both models already operate in a regime, where improvements to be made are only marginal.

Revised: Next, we describe the experimental details.

Original:  2 we train the SwinIR for color denoising from Liang et al.
Revised: To obtain Fig. 2 we train the SwinIR for color denoising from Liang et al.

Original: (2021) on the same training sets mentioned in Appx.
Revised: (2021) on the same training sets mentioned in Appx.

Original: A.1 with { 0 . 1 , 0 . 3 , 1 , 10 , 100 } thousand images from ImageNet.
Revised: A.1 with { 0 . 1 , 0 . 3 , 1 , 10 , 100 } thousand images from ImageNet.

Original: Instead of center cropping the training images to 256 × 256 we have to crop to 128 × 128 pixels as larger input patches would make it computational infeasible for us to train large versions of the SwinIR.
Revised: Instead of center cropping the training images to 256 × 256 we have to crop to 128 × 128 pixels as larger input patches would make it computational infeasible for us to train large versions of the SwinIR.

Original: The largest SwinIR alone took over 2 months to train on4 NVIDIA A40 GPUs.
Revised: The largest SwinIR alone took over 2 months to train on NVIDIA A40 GPUs.

Original: We train 4 different network sizes with { 3 . 7 , 11 . 5 , 41 . 8 , 128 . 9 } million parameters.
Revised: We train 4 different network sizes with { 3 . 7 , 11 . 5 , 41 . 8 , 128 . 9 } million parameters.

Original: We denote the four network sizes as small(S)/middle(M)/large(L)/huge(H).
Revised: We denote the four network sizes as small(S)/middle(M)/large(L)/huge(H).

Original: The training details and network configurations are as follows.
Revised: The training details and network configurations are as follows.

Original: The default SwinIR for denoising (Liang et al., 2021) was proposed for a training set size of about 10k images, 11.5M network parameters and was trained with a batch size of 8 for T = 1280 epochs, where the learning rate is halved at [0.5 T ,0.75 T , 0.875 T , 0.9375 T ] epochs.
Revised: The default SwinIR for denoising (Liang et al., 2021) was proposed for a training set size of about 10k images, 11.5M network parameters and was trained with a batch size of 8 for T = 1280 epochs, where the learning rate is halved at [0.5 T ,0.75 T , 0.875 T , 0.9375 T ] epochs.

Original: We keep the learning rate schedule but adjust the maximal number of epochs T according to the training set size.
Revised: We keep the learning rate schedule but adjust the maximal number of epochs T according to the training set size.

Original: Table 1 shows batch size and maximal number of epochs for every experiment in Fig.
Revised: Table 1 shows batch size and maximal number of epochs for every experiment in Fig.

Original: We did not optimize over the choice of the batch size but picked the batch size as prescribed by the availability of computational resources.[Table]
Revised: We did not optimize over the choice of the batch size but picked the batch size as prescribed by the availability of computational resources.

Original: See Liang et al.
Revised: See Liang et al.

Original: (2021) for a detailed description of the SwinIR network architecture.
Revised: (2021) for a detailed description of the SwinIR network architecture.

Original: We vary the network size by adjusting the number of residual Swin Transformer blocks, the number of Swin
Revised: We vary the network size by adjusting the number of residual Swin Transformer blocks, the number of Swin

Original: Transformers per block, the number of attention heads per Swin Transformer, the number of channels in the input embedding and the width of the fully connected layers in a Swin Transformer.
Revised: Transformers per block, the number of attention heads per Swin Transformer, the number of channels in the input embedding and the width of the fully connected layers in a Swin Transformer.

Original: Tablecontains a summary of the settings.
Revised: Tablecontains a summary of the settings.

Original: When scaling up the network size, we invested in the parameters that seemed to be most promising in the ablation studies in Liang et al.
Revised: When scaling up the network size, we invested in the parameters that seemed to be most promising in the ablation studies in Liang et al.

Original: The experiments were conducted on four NVIDIA A40 and four NVIDIA RTX A6000.
Revised: The experiments were conducted on four NVIDIA A40 and four NVIDIA RTX A6000.

Original: We used about 6600 GPU hours for the tranformer experiments in Fig.
Revised: We used about 13000 GPU hours for the tranformer experiments in Fig.

Original: For training the models in parallel on multiple GPUs we utilize the torch.distributed package with the glow backend, instead of the faster nccl backend, which was unfortunately not available on our hardware at that time. 
Revised: For training the models in parallel on multiple GPUs we utilize the torch.distributed package with the glow backend, instead of the faster nccl backend, which was unfortunately not available on our hardware at that time. [Table]

Original: B B ENCHMARKING OUR MODELS FOR G AUSSIAN IMAGE DENOISING
Revised: B B ENCHMARKING OUR MODELS FOR G AUSSIAN IMAGE DENOISING

Original: In this Section, we evaluate the models we trained for image denoising in Section 3 on four common test sets form the literature and show that the largest SwinIR trained on the largest dataset achieves new SOTA for all four test sets and the considered noise level.
Revised: In this Section [Table] we evaluate the models we trained for image denoising in Section 3 on four common test sets form the literature and show that the largest SwinIR trained on the largest dataset achieves new SOTA for all four test sets and the considered noise level.

Original: In Fig.
Revised: In Fig.

Original: 2 in the main body we compared the performance of the U-Nets and the SwinIRs trained on subsets of ImageNet for image denoising.
Revised: 2 in the main body we compared the performance of the U-Nets and the SwinIRs trained on subsets of ImageNet for image denoising.

Original: The models are evaluated on a test set sampled from
Revised: The models are evaluated on a test set sampled from

Original: ImageNet.
Revised: ImageNet.

Original: We observed that while SwinIRs significantly outperform U-Nets, the performance gain from increasing the training set size slows already at moderate training set sizes for both architectures equally.
Revised: We observed that while SwinIRs significantly outperform U-Nets, the performance gain from increasing the training set size slows already at moderate training set sizes for both architectures equally.

Original: However, there is still a moderat performance gain in scaling the models, and thus we expect the largest SwinIR trained on the largest dataset to outperform the original SwinIR from Liang et al.
Revised: However, there is still a moderat performance gain in scaling the models [Table] and thus we expect the largest SwinIR trained on the largest dataset to outperform the original SwinIR from Liang et al.

Original: Our results in this section show that this is indeed the case.
Revised: Our results in this section show that this is indeed the case.

Original: In Table 3 we evaluate on the standard test sets for Gaussian color image denoising CBSD

Revised: In Table 3 we evaluate on the standard test sets for Gaussian color image denoising CBSD

Original: (Martin et al., 2001), Kodak24 (Franzen, 1999), McMaster (Zhang et al., 2011) and Urban100 (Huang et al., 2015).
Revised: (Martin et al., 2001), Kodak24 (Franzen, 1999), McMaster (Zhang et al., 2011) and Urban100 (Huang et al., 2015).

Original: We observe a significant performance difference between the best U-Net (46.5M parameters, 100k training images) and the other transformer based methods.
Revised: We observe a significant performance difference between the best U-Net (46.5M parameters, 100k training images) and the other transformer based methods.

Original: As expected, our largest
Revised: As expected, our largest

Original: SwinIR trained on the largest dataset SwinIR 100/H outperforms the original SwinIR, but also the
Revised: SwinIR trained on the largest dataset SwinIR 100/H outperforms the original SwinIR, but also the

Original: SCUnet (Zhang et al., 2022) a later SOTA model that has been demonstrated to outperform the original SwinIR.
Revised: SCUnet (Zhang et al., 2022) a later SOTA model that has been demonstrated to outperform the original SwinIR.

Original: We also depict the gains for the SwinIR from just scaling up the network size and then from scaling up network size and training set size.
Revised: We also depict the gains for the SwinIR from just scaling up the network size and then from scaling up network size and training set size.

Original: While this led to a new SOTA for Gaussian image denoising, note that on the downside training the SwinIR 10/M, which is comparable to the original SwinIR, took about 2 weeks on 4 NVIDIA A40 gpus, while training the SwinIR 100/H took over 2 months.
Revised: While this led to a new SOTA for Gaussian image denoising, note that on the downside training the SwinIR 10/M, which is comparable to the original SwinIR, took about 2 weeks on 4 NVIDIA A40 gpus, while training the SwinIR 100/H took over 2 months.

Original: C E MPIRICAL SCALING LAWS FOR IMAGE SUPER - RESOLUTION WITH A U-N ET
Revised: C E MPIRICAL SCALING LAWS FOR IMAGE SUPER - RESOLUTION WITH A U-N ET

Original: In this section, we consider the problem of super-resolution, i.e., estimating an high-resolution image from a low-resolution version of the image.
Revised: In this Section, we consider the problem of super-resolution, i.e., estimating an high-resolution image from a low-resolution version of the image.

Original: This can be viewed as a compressive sensing problem, since we can view the super-resolution problem as reconstructing a signal x ∈ R n from a downsampled version y
Revised: This can be viewed as a compressive sensing problem, since we can view the super-resolution problem as reconstructing a signal x ∈ R n from a downsampled version y

Original: =
Revised: =

Original: Ax , where the matrix A ∈ R m × n implements a downsampling operation[Table] like bicubic downsampling, or blurring followed by downsampling.
Revised: Ax , where the matrix A ∈ R m × n implements a downsampling operation like bicubic downsampling, or blurring followed by downsampling.

Original: As first shown in the pioneering work of Dong et al.
Revised: As first shown in the pioneering work of Dong et al.

Original: (2014) data driven neural networks trained end-to-end outperform classical model-based approaches (Gu et al., 2012; Michaeli & Irani, 2013; Timofte et al., 2013).
Revised: (2014) data driven neural networks trained end-to-end outperform classical model-based approaches (Gu et al., 2012; Michaeli & Irani, 2013; Timofte et al., 2013).

Original: Dong et al.
Revised: Dong et al.

Original: (2014) reports that for super-resolution with a simple three-layer CNN, the gains from very large training sets do not seem to be as impressive as in high-level vision problems like image classification.
Revised: (2014) reports that for super-resolution with a simple three-layer CNN, the gains from very large training sets do not seem to be as impressive as in high-level vision problems like image classification.

Original: Liang et al.
Revised: Liang et al.

Original: (2021) plot the super-resolution performance of the SOTA SwinIR model as a function of the training set size up to 3600 training examples for a fixed network size.
Revised: (2021) plot the super-resolution performance of the SOTA SwinIR model as a function of the training set size up to 3600 training examples for a fixed network size.

Original: When plotting their results on a logarithmic scale, we observe that the performance improvement follows a power-law.
Revised: When plotting their results on a logarithmic scale, we observe that the performance improvement follows a power-law.

Original: It is unclear, however, whether this power law slows beyond this relatively small number of images.
Revised: It is unclear, however, whether this power law slows beyond this relatively small number of images.

Original: In this Section, we obtain scaling laws for super-resolution for a U-Net over a wide range of training set and network sizes, similar as for denoising and compressive sensing in the main body.
Revised: In this Section, we obtain scaling laws for super-resolution for a U-Net over a wide range of training set and network sizes, similar as for denoising and compressive sensing in the main body.

Original: Datasets.
Revised: Datasets.

Original: We use the same training, validation and test sets as described in Appx.
Revised: We use the same training, validation and test sets as described in Appx.

Original: A.1 with training set sizes N ∈ [100 , 100000] images from ImageNet.
Revised: A.1 with training set sizes N ∈ [100 , 100 k ] images from ImageNet.

Revised: On top we add two larger training sets of size 300k and 600k.

Original: Instead of center cropping the training images to 256 × 256 we follow the super-resolution experiments in Liang et al.
Revised: Instead of center cropping the training images to 256 × 256 we follow the super-resolution experiments in Liang et al.

Original: and train on images cropped to 128 × 128 pixels.
Revised: and train on images cropped to 128 × 128 pixels.

Original: We consider super-resolution of factor 2 so the low-resolution images have size 64 × 64 .
Revised: We consider super-resolution of factor 2 so the low-resolution images have size 64 × 64 .

Original: The low-resolution images are obtained with the bicubic downsampling function of Python’s PIL.Image package.
Revised: The low-resolution images are obtained with the bicubic downsampling function of Python’s PIL.Image package.

Original: Model variants and training.
Revised: Model variants and training.

Original: We train the same U-Net model used in Section 3 and described in
Revised: We train the same U-Net model used in Section 3 and described in

Original: Appx.
Revised: Appx.

Original: A.1 but with only one block per encoder/decoder.
Revised: A.1 but with only one block per encoder/decoder as this resulted in slightly better results than with two blocks.

Original: The network is trained end-to-end to map a coarse reconstruction, obtained through bicubic upsampling, to the residual between the coarse reconstruction and the high-resolution ground truth image.
Revised: The network is trained end-to-end to map a coarse reconstruction, obtained through bicubic upsampling, to the residual between the coarse reconstruction and the high-resolution ground truth image.

Original: We vary the number of the channels in the first layer in { 8 , 16 , 32 , 64 , 128 , 192 , 256 , 320 , 448  } , which corresponds to { 0 . 007 , 0 . 025 , 0 . 10 , 0 .
Revised: We vary the number of the channels in the first layer in { 8 , 16 , 32 , 64 , 128 , 192 , 256 , 320 , 448 , 564 } , which corresponds to { 0 . 007 , 0 . 025 , 0 . 10 , 0 .

Original: 40 , 1 . 6 , 3 . 6 , 6 . 4 , 10 . 0 , 20 . 0  } million network parameters.
Revised: 40 , 1 . 6 , 3 . 6 , 6 . 4 , 10 . 0 , 20 . 0 , 31 . 2 } million network parameters.

Original: We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.
Revised: We do not use any data augmentation, since it is unclear how to account for it in the number of training examples.

Original: We use the ℓ 1 -loss and Adam optimizer with its default settings.
Revised: We use the ℓ 1 -loss and Adam optimizer with its default settings.

Original: For all experiments we find a good initial learning rate with the same annealing strategy as described in Appx.
Revised: For all experiments we find a good initial learning rate with the same annealing strategy as described in Appx.

Original: A.1.
Revised: A.1.

Original: However, instead of picking the largest learning rate for which the validation loss does not diverge, we pick the second largest, which leads to slightly more stable results.
Revised: However, instead of picking the largest learning rate for which the validation loss does not diverge, we pick the second largest, which leads to slightly more stable results.

Original: We start the annealing with learning rate of 10 − 5 .
Revised: We start the annealing with learning rate of 10 − 5 .

Original: In the few cases, where our heuristic leads to a degenerated training curve, typically due to picking a significantly too small or too large learning rate, starting the annealing with a smaller learning rate of 10 − 6 resolves the problem.
Revised: In the few cases, where our heuristic leads to a degenerated training curve, typically due to picking a significantly too small or too large learning rate, starting the annealing with a smaller learning rate of 10 − 6 resolves the problem.

Original: We do not put a limit on the amount of compute invested into training.
Revised: We do not put a limit on the amount of compute invested into training.

Original: To this end, we deploy an automated learning rate decay that reduces the learning rate by 0 .
Revised: To this end, we deploy an automated learning rate decay that reduces the learning rate by 0 .

Original: 5 if the validation PSNR has not improved by at least 0.001 for 8 epochs.
Revised: 5 if the validation PSNR has not improved by at least 0.001 for 8 epochs.

Original: We stop the training once the validation loss did not improve for two consecutive learning rates.
Revised: We stop the training once the validation loss did not improve for two consecutive learning rates.

Original: For training sets up to 10000 images we found that batch size ofworks best.
Revised: For training sets up to 10000 images we found that batch size ofworks best.

Original: For larger training sets we use a batch size of 10.
Revised: For larger training sets we use a batch size of 10.

Original: [Figure]

Original: Colors in the correspond to the same training set size.

Original: While the scaling laws for super-resolution do not slow as early as those for Denoising and Compressive sensing (see

Original: Fig.1), they are likely to slow further at a larger number of training examples

Original: For training set sizes up to 10000 we train 3 random seeds and pick the best run.
Revised: For training set sizes up to 10000 we train 3 random seeds and pick the best run.

Original: As the variance between runs compared to the gain in performance between different training set sizes decreases with increasing training set size we only run one seed for larger training set sizes.
Revised: As the variance between runs compared to the gain in performance between different training set sizes decreases with increasing training set size we only run one seed for larger training set sizes.

Original: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA
Revised: The experiments were conducted on four NVIDIA A40, four NVIDIA RTX A6000 and four NVIDIA

Original: Quadro RTX 6000 GPUs.
Revised: Quadro RTX 6000 GPUs.

Original: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 430 GPU hours for the experiments in Fig.
Revised: We measure the time in GPU hours until the best epoch according to the validation loss resulting in about 1500 GPU hours for the experiments in Fig.

Original: Results and discussion.
Revised: Results and discussion.

Original: For each training set size, Fig. 5(a) shows the reconstruction performance in PSNR of the best model over all simulated network sizes in Fig. 5(b).
Revised: For each training set size, Fig. 6(a) shows the reconstruction performance in PSNR of the best model over all simulated network sizes in Fig. 6(b).

Original: Since the curves per training set size in Fig.
Revised: Since the curves per training set size in Fig.

Original: 5(b) are relatively flat, further scaling up the network size is not expected to significantly improve the performance on the studied training sets.
Revised: 6(b) are relatively flat, further scaling up the network size is not expected to significantly improve the performance on the studied training sets.

Original: Here are the two main findings:
Revised: Here are the two main findings:

Original: A linear power law with a scaling coefficient α = 0 . 0077 holds roughly up to training set sizes of about 10k images, and for training set sizes starting from 10k this slows to a linear power law with  smaller scaling coefficient α = 0 . 
Revised: A linear power law with a scaling coefficient α = 0 . 0075 holds roughly up to training set sizes of about 30k images, and for training set sizes starting from 60k this slows to a linear power law with significantly smaller scaling coefficient α = 0 . 0029 .

Original: While the scaling laws for super-resolution do not slow as early as those for denoising and compressive sensing (see Fig.1), they are likely to slow further at a larger number of training examples.

Original: Even if the scaling law were not to slow further relatively small gains are expected from scaling the training data further:

Original: Increasing the training set sizefrom 10k by three orders of magnitudes is expected to increase performance by about 1dB, at most (assuming the relation 31 .
Revised: With this slowed scaling law a training set size of 1.6B images would be required to increase performance by another 1dB (assuming the relation 32 .

Original: 13 ∗ N 0 .
Revised: 05 N 0 .

Original: 0055 persists, which is likely to slow down even further).
Revised: 0029 persists, which is likely to slow down even further).

Revised: While slowing already at a few tens of thousands of training images, the scaling laws for superresolution do not slow as early as those for denoising and compressive sensing (see Fig.1).

Revised: This could be partially due training on image patches of size 128 × 128 as opposed to size 256 × 256 used for denoising.

Revised: [Figure]

Original:  D E MPIRICAL SCALING LAWS FOR  DENOISINGWITH FIXED NOISE
Revised: Colors in correspond to the same While the scaling laws for super-resolution do not slow as early as those for Denoising and Compressive sensing (see Fig.1), they are likely to slow further at a larger number of training examples D A DDITIONAL EMPIRICAL SCALING LAWS FOR G AUSSIAN DENOISING

Revised: In this Section, we extend our results for Gaussian denoising with a U-Net from Section 3 with two additional setups.

Revised: Section D.1 considers fixing the noise sampled in each training epoch and

Revised: Section D.2 investigates reducing the noise level from σ z = 25 to σ z = 15 .

Revised: D. E MPIRICAL SCALING LAWS FOR DENOISING WITH FIXED NOISE

Original: Our main results for denoising with a U-Net trained end-to-end discussed in Section 3 follow a setup in which the noise per training example is re-sampled in every training epoch.
Revised: Our main results for denoising with a U-Net trained end-to-end discussed in Section 3 follow a setup in which the noise per training example is re-sampled in every training epoch.

Original: We choose this mode since it makes the best use of the available clean images.
Revised: We choose this setup since it makes the best use of the available clean images.

Original: However, fixing the noise is also interesting since it is perhaps more similar to a  setup in which the noise statistics are unknown like in  real-world noise removal.
Revised: However, fixing the noise is also interesting since it is closer to a denoising setup in which the noise statistics are unknown, which is the case in some real-world noise removal problems.

Revised: In such a problem, we would be given pairs of noisy and clean image, and could not synthesis new noisy images from the clean images.

Original: In this section, we follow the same experimental setup from Appx.
Revised: In this section, we follow the same experimental setup from Appx.

Original: A.1 to simulate the performance of a U-Net for denoising with fixed noise for up to 100k training images (see Fig. 6).
Revised: A.1 to simulate the performance of a U-Net for denoising with fixed noise for up to 100k training images (see Fig. 7).

Original: Compared to re-sampling the noise, we observe a drop in performance of about 0.3dB for small training set sizes.
Revised: Compared to re-sampling the noise, we observe a drop in performance of about 0.3dB for small training set sizes.

Original: The performance difference at 10k images is reduced to 0.2dB resulting in a slightly steeper scaling law for moderate training set sizes.
Revised: The performance difference at 10k images is reduced to 0.2dB resulting in a slightly steeper scaling law for moderate training set sizes.

Original: However, at around 10k training images the scaling of the performance of training with fixed noise also starts to flatten as it approaches the performance of re-sampling the noise during training.
Revised: However, at around 10k training images the scaling of the performance of training with fixed noise also starts to flatten as it approaches the performance of re-sampling the noise during training.

Original: This shows that if the noise statistics are unknown, more data is required to achieve the same performance as when they are known.
Revised: This indicates that if the noise statistics are unknown, more data is required to achieve the same performance as when they are known.

Original: However, in both cases the scaling with training set size slows already down at moderate training set sizes.
Revised: However, in both cases the scaling with training set size slows already down at moderate training set sizes.

Revised: D. E MPIRICAL SCALING LAWS FOR DENOISING WITH A SMALLER NOISE LEVEL

Revised: Our results for Gaussian denoising in Section 3 are for a fixed noise level of σ z = 25 .

Revised: In this Section, we repeat the experiments for Gaussian denoising with a U-Net described in Appx.

Revised: A.1 with smaller noise level of σ z = 15 , in order to see how the scaling laws change.

Original: [Figure]
Revised: The results for both noise levels are depicted in Fig. 8. Note that the curves for σ z = 15 in (a),(b) are preliminary as we are still waiting for the training of the larger training set sizes to terminate. We observe an improvement of about 1.3dB in PSNR, which is expected since the irreducible error decreases for smaller noise levels. We also observe that the scaling coefficient for the smaller noise level σ z = 15 (i.e., α = 0 . 0027 ) is slightly stepper than that for the larger noise level σ z = 25 (i.e., α = 0 . 0019 ).

Revised: This coincides with the qualitative behavior of the curves for subspace denoising in

Revised: Figure 3.

Revised: Apart from that the curves are qualitatively similar, in that a initially steep power law is replaced by a slower one at around 6000 training images.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: N Number of training patches of size 128 ×

Original: [Figure]
Revised: [Figure]

Revised: Fix: 30 .

Revised: 62 N 0 .

Revised: Fix: 31 .

Revised: 65 N 0 .101010 31 .31 .31 .31 .

Revised: 32 .32 .(b)

Revised: Network parameters P

Revised: D. E MPIRICAL SCALING LAWS FOR DENOISING WITH A SMALLER PATCH SIZE

Revised: Our results for Gaussian denoising in Section 3 with a U-Net were obtained for a constant training patch size of 256 × 256 pixels across all network and training set sizes.

Revised: In this Section, we repeat the experiments for Gaussian denoising with a U-Net described in Appx.

Revised: A.1 with a smaller patch size of 128 × 128 pixels.

Revised: The results for both patch sizes are depicted in Fig.

Revised: We observe that in the regime of large training set sizes, that we are primarily interested in, training on N patches of size 256 × 256 is more beneficial than training on 4 N patches of size 128 × 128 .

Revised: We therefore focus on patch size 256 × 256 in the main body of this work.

Revised: E U NDERSTANDING SCALING LAWS FOR DENOISING THEORETICALLYSUPPLEMENTARY RESULTS

Revised: In this section, we provide additional details on the statements in Section 5 on understanding scaling laws for denoising theoretically by studying a linear subspace denoising problem theoretically, and provide additional numerical results.

Revised: Recall that we consider a linear estimator of the form f W ( y )

Revised: =

Revised: Wy , and measure performance in terms of the expected mean-squared reconstruction error (normalized by the latent signal dimension d ):

Revised: [Equation]

Revised: Above, expectation is over the joint distribution of ( x , y ) , and the second equality follows from using that x

Revised: = Uc , where c ∼ N (0 , I ) is Gaussian, and y

Revised: = x

Revised: + z , where the noise z ∼ N (0 , σ 2 z I ) is

Revised: Gaussian.

Revised: The optimal linear estimator.

Revised: The optimal linear estimator (i.e., the estimator that minimizes the risk defined in equation (1)) is given by W ∗ = 1 1+ σ 2 z UU T .

Revised: This follows from taking the gradient of the risk (1), setting it to zero, and solving for W .

Revised: The estimator projects the data onto the subspace and shrinks towards zero, depending on the noise variance.

Revised: The associated risk is

Revised: R ( W ∗ )

Revised: =

Revised: σ 2 z / (1 + σ 2 z ) .

Revised: Early-stopped empirical risk minimization.

Revised: We consider the estimator that applies gradient descent to the empirical risk

Revised: [Equation]

Revised: where X , Y ∈ R n × N contain the training examples as columns, and early-stops after k iterations for regularization.

Revised: We next discuss the early-stopped estimator W k in more detail.

Revised: Fig.

Revised: 10 numerically demonstrates the regularizing effect of early stopping gradient descent, where W ∞ =

Revised: XY † is the converged learned estimator (see Appx.

Revised: G, Eq.

Revised: We see that regularization is necessary for this estimator to perform well.

Revised: We next discuss Theorem 2 and the associated assumptions in more detail.

Revised: Theorem considers the following regime: (i) the number of training examples obeys ( d + nσ 2 z ) log( n ) ≤ N and (ii)

Revised: N ≤ ξd/σ 2 z , for an arbitrary ξ , and (iii) N log( N ) ≤ n .

Revised: For this regime, the theorem guarantees that the risk of the optimally early-stopped estimator obeys, with high probability,

Revised: [Equation]

Revised: The theorem looks similar to that for the PCA estimate (Theorem 1), in that the risk is a constant away from the optimal risk, with an error term that becomes small as ( d + nσ 2 z ) /N becomes small.

Revised: However, the error bound does not converge to R ( W ∗ ) as the number of training examples, N , converges to infinity.

Revised: This is probably an artifact of our analysis, but it is unclear, at least to us, how to derive a substantially tighter bound.

Revised: In our analysis (see appendix G), we balance two errors: One error decreases in k and is associated with the part of the signal projected into the subspace, and the second error increases with k and is associated with the orthogonal complement of the subspace.

Revised: We choose the early-stopping time to optimally balance those two terms, which yields the stated bound (3).

Revised: Now with regards to the assumption: Assumption (iii) N log( N ) ≤ n means we are in the highdimensional regime; we think this is somewhat closer to reality (for example for denoising a 512 ×image, this would require the number of training examples to be smaller than 250 k), but we can derive an analogous bound for the regime N log( N ) ≥ n , where the number of training examples is larger than the ambient dimension.

Revised: Assumption (i) ( d + nσ 2 z ) log( n ) ≤ N is relatively mild, as it is necessary to being able to somewhat accurately estimate the subspace; this assumption is also required for the PCA estimate.

Revised: Assumption (ii) N ≤ ξd/σ 2 z , for an arbitrary ξ , is not restrictive in that ξ can be arbitrarily large, we make this assumption only so that the theorem can be stated in a convenient way.

Revised: However, assumption (ii) reveals a shortcoming of Theorem 2 which is that we cannot make the bound go to zero as N → ∞ , since increasing ξ increases one term in the bound, and decreases another one.

Original: E A DDITIONAL NUMERICAL SIMULATIONS
Revised: E. A DDITIONAL NUMERICAL SIMULATIONS

Original: In this Section we provide further numerical simulations for the PCA subspace estimator and the estimator learned with early stopped gradient descent discussed in Section 5, Theorem 1 and Theorem2.
Revised: In this Section we provide further numerical simulations for the PCA subspace estimator and the estimator learned with early stopped gradient descent discussed in Section 5, Theorem 1 and Theorem2.

Original: Similar to Fig.
Revised: Similar to Fig.

Original: 4, Fig.
Revised: 3, Fig.

Original: 7 shows the risks R ( W k opt ) and R ( W PCA ) as a function of the number of training examples N for varying values of the signal and ambient dimension d and n , while fixing all other model parameters.
Revised: 11 shows the risks R ( W k opt ) and R ( W PCA ) as a function of the number of training examples N for varying values of the signal and ambient dimension d and n , while fixing all other model parameters.

Original: In the power law region we fit linear power laws with negative scaling coefficients α .
Revised: In the power law region we fit linear power laws with negative scaling coefficients α .

Original: We observe steeper power laws (larger | α | ) for smaller ambient dimensions n and larger signal dimensions d .
Revised: We observe steeper power laws (larger | α | ) for smaller ambient dimensions n and larger signal dimensions d .

Original: Also the scaling coefficients of the learned estimator consistently excel the coefficients from the PCA estimator.
Revised: Also the scaling coefficients of the learned estimator consistently excel the coefficients from the PCA estimator.

Original: F P ROOF FOR T HEOREM 1: R ISK BOUND FOR PCA SUBSPACE ESTIMATION
Revised: F P ROOF FOR T HEOREM 1: R ISK BOUND FOR PCA SUBSPACE ESTIMATION

Original: We provide a bound on the risk of the estimator f ( y ) =
Revised: We provide a bound on the risk of the estimator f ( y ) =

Original: W PCA y with W PCA =
Revised: W PCA y with W PCA =

Original: τ ˆ U
Revised: τ ˆ U

Original: ˆ U T and τ = 1 1+ σ 2 z .
Revised: ˆ U T and τ = 1 1+ σ 2 z .

Original: Recall that ˆ U ∈ R n × d contains the singular vectors corresponding to the d -leading singular values of YY T .
Revised: Recall that ˆ U ∈ R n × d contains the singular vectors corresponding to the d -leading [Figure] [Figure] with varying model parameters the scaling coefficients α change.

Revised: Further, the learned estimator exhibits steeper scaling than the PCA estimator over all settings.

Revised: Error bars are over 5 independent runs.

Original:  We define ˆ U ⊥ ∈ R n × n − d as the orthogonal complement of ˆ U .
Revised: singular values of YY T . We define ˆ U ⊥ ∈ R n × n − d as the orthogonal complement of ˆ U .

Original: Starting from the risk expression given in equation (1) we obtain
Revised: Starting from the risk expression given in equation (1) we obtain

Original: [Equation]
Revised: [Equation]

Original: Here, inequality (ii) follows from Section H.1 equation (33) and holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − n − 10 − 3 e − d + e − n .
Revised: Here, inequality (ii) follows from Section H.1 equation (34) and holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − n − 10 − 3 e − d + e − n .

Original: This concludes the proof.
Revised: This concludes the proof.

Original: G P ROOF OF T HEOREM 2: R ISK BOUND FOR EARLY STOPPED EMPIRICAL RISK MINIMIZATION
Revised: G P ROOF OF T HEOREM 2: R ISK BOUND FOR EARLY STOPPED EMPIRICAL RISK MINIMIZATION

Original: This section contains the proof ofour main result in Theorem 2.
Revised: This section contains the proof of Theorem 2.

Original: The theorem characterizes the performance of the learned, linear estimator W k that is obtained by applying k iterations of gradient descent with stepsize η starting at W 0 = 0 to the loss L ( W ) in equation (2).
Revised: The theorem characterizes the performance of the learned, linear estimator W k that is obtained by applying k iterations of gradient descent with stepsize η starting at W 0 = 0 to the loss L ( W ) in equation (2).

Original: We start by deriving a closed form expression for the estimator W k .
Revised: We start by deriving a closed form expression for the estimator W k .

Original: The gradient of the loss is
Revised: The gradient of the loss is

Original: [Equation]
Revised: [Equation]

Original: and thus the iterations of gradient descent are
Revised: and thus the iterations of gradient descent are

Original: [Equation]
Revised: [Equation]

Original: Let Y =
Revised: Let Y =

Original: U y Σ
Revised: U y Σ

Original: y V Ty ∈ R n × N and X
Revised: y V Ty ∈ R n × N and X

Original: =
Revised: =

Original: U x Σ x V Tx ∈ R n × N be the singular value decompositions of Y and X respectively and assume that the singular values are non-zero and descending, i.e., σ y, 1 ≥ σ y, 2 ≥ . . . .
Revised: U x Σ x V Tx ∈ R n × N be the singular value decompositions of Y and X respectively and assume that the singular values are non-zero and descending, i.e., σ y, 1 ≥ σ y, 2 ≥ . . . .

Original: We have, with W 0 = 0 and k ≥ 1 that
Revised: We have, with W 0 = 0 and k ≥ 1 that

Original: [Equation]
Revised: [Equation]

Original: where we defined D k ∈ R N × N as a diagonal matrix with i -th diagonal entry given by (1 − (1 − ησ 2 y,i ) k )
Revised: where we defined D k ∈ R N × N as a diagonal matrix with i -th diagonal entry given by (1 − (1 − ησ 2 y,i ) k )

Original: /σ
Revised: /σ

Original: y,i and where we used the geometric series to obtain
Revised: y,i and where we used the geometric series to obtain

Original: [Equation]
Revised: [Equation]

Original: Note that for k → ∞ and choosing η such that 1 − ησ 2 y,i < 1 for all i we get
Revised: Note that for k → ∞ and choosing η such that 1 − ησ 2 y,i < 1 for all i we get

Original: [Equation]
Revised: [Equation]

Original: Evaluating the risk from equation (1) at the estimator W k gives
Revised: Evaluating the risk from equation (1) at the estimator W k gives

Original: [Equation]
Revised: [Equation]

Original: To shorten notation we define
Revised: To shorten notation we define

Original: [Equation]
Revised: [Equation]

Original: [Equation] We next provide bounds for the two terms on the right-hand-side of equation (5), proven later in this section. 
Revised: We next provide bounds for the two terms on the right-hand-side of equation (6), proven later in this section. Bound on the first term in equation (6) :

Original: In Section G.1 we show that provided N log( N ) ≤ n , 9 d ≤ N and ( d + nσ 2 z ) log( n ) ≤ N , for some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N − 2 e − n/ 2 , the following bound holds:
Revised: In Section G.1 we show that provided N log( N ) ≤ n , 9 d ≤ N and ( d + nσ 2 z ) log( n ) ≤ N , for some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N − 2 e − n/ 2 , the following bound holds:

Original: [Equation]
Revised: [Equation]

Original:  In Section G.2 we show
Revised: Bound on the second term in equation (6) : In Section G.2 we show

Original: [Equation]
Revised: [Equation]

Original: With equations(8) and (9)  in place and by splitting up the sum in (9) we can bound the right hand side of equation (5) as
Revised: With equations (9) and (10) in place and by splitting up the sum in (10) we can bound the right hand side of equation (6) as

Original: [Equation]
Revised: [Equation]

Original: Since the singular values of Y are in descending order σ y, 1 ≥ σ y, 2 ≥ . . 
Revised: Since the singular values of Y are in descending order σ y [Equation] 1 ≥ σ y, 2 ≥ . . . [Equation]

Original: , this is, for any iteration k , bounded as
Revised: this is, for any iteration k , bounded as

Original: [Equation]
Revised: [Equation]

Original: This is a good bound if we’re at an iteration sufficiently large so that (1 − ησ 2 y, 1 ) k is small.
Revised: This is a good bound if we’re at an iteration sufficiently large so that (1 − ησ 2 y, 1 ) k is small.

Original: In (11) we have (1 − ησ 2 y,d ) 2 k that is decreasing in the number of gradient descent steps k and also decreasing in the stepsize η as long as ησ 2 y,i ≤ 1 for i = 1 , . . .
Revised: In (12) we have (1 − ησ 2 y,d ) 2 k that is decreasing in the number of gradient descent steps k and also decreasing in the stepsize η as long as ησ 2 y,i ≤ 1 for i = 1 , . . .

Original: , d .
Revised: , d .

Original: Further, we have (1 − (1 − ησ 2 y,i ) k )that is increasing in k and also increasing in η .
Revised: Further, we have (1 − (1 − ησ 2 y,i ) k )that is increasing in k and also increasing in η .

Original: The first term corresponds to the signal that we want to fit sufficiently well, whereas the second term corresponds to the noise from which we want to fit as little as possible.
Revised: The first term corresponds to the signal that we want to fit sufficiently well, whereas the second term corresponds to the noise from which we want to fit as little as possible.

Original: Hence, there exist optimal choices for k, η that trade-off the sum of the two terms.
Revised: Hence, there exist optimal choices for k, η that trade-off the sum of the two terms.

Original: In our setup the d leading singular values of Y corresponding to the signal are large and concentrate around N (1 + σ 2 z ) , while the remaining singular values are small and concentrate around Nσ 2 z .
Revised: In our setup the d leading singular values of Y corresponding to the signal are large and concentrate around N (1 + σ 2 z ) , while the remaining singular values are small and concentrate around Nσ 2 z .

Original: Hence, we can apply a single step of gradient descent k = 1 to already fit a large portion of the signal, while minimizing the portion of the noise that is fitted.
Revised: Hence, we can apply a single step of gradient descent k = 1 to already fit a large portion of the signal, while minimizing the portion of the noise that is fitted.

Original: For that we choose the stepsize η as large as possible such that ησ 2 y,i ≤ 1 for i = 1 , . . .
Revised: For that we choose the stepsize η as large as possible such that ησ 2 y,i ≤ 1 for i = 1 , . . .

Original: , d still holds.
Revised: , d still holds.

Original: Next, suppose the following events hold
Revised: Next, suppose the following events hold

Original: [Equation]
Revised: [Equation]

Original: In Section H.

Original: [Equation]
Revised: [Equation]

Original:  we show that[Equation] In Section H.
Revised: In Section H.4, we show that

Original: [Equation]
Revised: [Equation]

Original:  we also show that, provided that N ≥ 

Revised: In Section H.4, we also show that, provided that N ≥

Original: Cϵ − 2 ( d + σ 2 z n ) log n , for some constant C and ϵ ∈ (0 ,1) ,
Revised: Cϵ − 2 ( d + σ 2 z n ) log n , for some constant C and ϵ ∈ (0 ,

Original: [Equation]
Revised: [Equation]

Original: We next bound the terms in equation (11).
Revised: We next bound the terms in equation (12).

Original: As discussed, we set k = 1 and the stepsize as large as possible, i [Equation] e. η = 1 / (cid:0) N (1 + ϵ  ) 
Revised: As discussed, we set k = 1 and the stepsize as large as possible, i.e. η = 1 / (cid:0) N (1 + ϵ )(1 + σ 2 z ) (cid:1) ≤ 1 /σ 2 y, 1 , which holds on event E 4 .

Original: [Equation]

Original: 1 + σ 2 z

Original: [Equation]

Original: (cid:1) ≤ 1 /σ 2 y, 1 , which holds on event E 4 .

Original: Finally, on event Ewe obtain
Revised: Finally, on event Ewe obtain

Original: [Equation]
Revised: [Equation]

Original: Next we bound the sum in equation (11).
Revised: Next we bound the sum in equation (12).

Original: Towards this goal [Equation] we upper bound each term in the sum with its linear approximation at the origin.
Revised: Towards this goal, we upper bound each term in the sum with its linear approximation at the origin.

Original: We compute the derivative at the origin as
Revised: We compute the derivative at the origin as

Original: [Equation]
Revised: [Equation]

Original: Thus, we have
Revised: Thus, we have

Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: on the event E 1 and for σ 2 z ≤ 1 , k = 1 and η = 1 / (cid:0) N (1 + ϵ )(1 + σ 2 z ) (cid:1) .
Revised: on the event E 1 and for σ 2 z ≤ 1 , k = 1 and η = 1 / (cid:0) N (1 + ϵ )(1 + σ 2 z ) (cid:1) .

Original: Putting this together and on the events E 2 , E 3 we get the bound 
Revised: Putting this together and on the events E 2 , E 3 we get the bound R

Original: [Equation]
Revised: [Equation]

Original: The bound holds provided 3 Cϵ − 2 ( d + σ 2 z n ) log n ≤ N and N log( N ) ≤ n , for some constants c, C and with probability at least 1 − 2 e − N/ 8 − 2 e − N/ 18 − 5 n − 9 − 5 e − d − 2 e − n − e − N − 2 e − n/ 2 .
Revised: The bound holds provided 3 Cϵ − 2 ( d + σ 2 z n ) log n ≤ N and N log( N ) ≤ n , for some constants c, C and with probability at least 1 − 2 e − N/ 8 − 2 e − N/ 18 − 5 n − 9 − 5 e − d − 2 e − n − e − N − 2 e − n/ 2 .

Original: To maximize the benefit from early stopping we set ϵ as small as possible with respect to the condition3 Cϵ
Revised: To maximize the benefit from early stopping we set ϵ as small as possible with respect to the condition

Original: [Equation]
Revised: [Equation]

Original: With that equation (18) becomes
Revised: With that equation (19) becomes

Revised: [Equation]

Revised: We now consider the regime where N ≤ ξ d σ 2 z , for a numerical constant ξ , to simplify the statement further.

Revised: For this regime, we have

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof of Theorem 2.
Revised: This concludes the proof of Theorem 2.

Original: G. P ROOF OF EQUATION (8)
Revised: G. P ROOF OF EQUATION (9)

Original: In this Section, we derive a bound for 1 d (cid:13) (cid:13) (
Revised: In this Section, we derive a bound for 1 d (cid:13)(cid:13)

Original:  XV y D k U
Revised: ( XV y D k U

Original: Ty − I ) U (cid:13) (cid:13) 2 F the first term in (5).
Revised: Ty − I ) U (cid:13)(cid:13) 2 F the first term in (6).

Original: To this end, we introduce further notation.
Revised: To this end, we introduce further notation.

Original: Recall that Y =
Revised: Recall that Y =

Original: U y Σ
Revised: U y Σ

Original: y V Ty ∈ R n × N and X
Revised: y V Ty ∈ R n × N and X

Original: =
Revised: =

Original: U x Σ x V Tx ∈ R n × N are the SVDs of Y and X respectively.
Revised: U x Σ x V Tx ∈ R n × N are the SVDs of Y and X respectively.

Original: All derivations below hold regardless of whether N ≤ n or N > n .
Revised: All derivations below hold regardless of whether N ≤ n or N > n .

Original: Exemplarily, we will show them for N ≤ n .
Revised: Exemplarily, we will show them for N ≤ n .

Original: Thus, U y ∈ R n × N , Σ y ∈ R N × N ,
Revised: Thus, U y ∈ R n × N , Σ y ∈ R N × N ,

Original: V y ∈ R N × N and U x ∈ R n × d , Σ x ∈ R d × d , V x ∈ R N × d .
Revised: V y ∈ R N × N and U x ∈ R n × d , Σ x ∈ R d × d , V x ∈ R N × d .

Original: Let U
Revised: Let U

Original: y 1 , U x 1 ∈ R n × d be the d leading left singular vectors of Y and X (note that U x 1 = U x ), let U y 2 , U x 2 ∈ R n × n − d be their orthonormal complements.
Revised: y 1 , U x 1 ∈ R n × d be the d leading left singular vectors of Y and X (note that U x 1 = U x ), let U y 2 , U x 2 ∈ R n × n − d be their orthonormal complements.

Original: Let ˜ U
Revised: Let ˜ U

Original: y
Revised: y

Original: =
Revised: =

Original: [ U y
Revised: [ U y

Original: U y 2 ] ∈ R n × n and ˜ U x
Revised: U y 2 ] ∈ R n × n and ˜ U x

Original: = [ U x
Revised: = [ U x

Original: U x 2 ] ∈ R n × n .
Revised: U x 2 ] ∈ R n × n .

Original: Analogous definitions can be made for the leading right singular vectors of Y and X .
Revised: Analogous definitions can be made for the leading right singular vectors of Y and X .

Original: Recall that D k = diag (cid:0) . . .
Revised: Recall that D k = diag (cid:0) . . .

Original: , (1 − (1 − ησ 2 y,i ) k ) /σ
Revised: , (1 − (1 − ησ 2 y,i ) k ) /σ

Original: y,i , . . .
Revised: y,i , . . .

Original: (cid:1) ∈ R N × N .
Revised: (cid:1) ∈ R N × N .

Original: Let ˜ D
Revised: Let ˜ D

Original: k =
Revised: k =

Original: [ D k 0 ]
Revised: [ D k 0 ]

Original: ∈ R N × n and define ˜ D k 1 ∈ R d × d and ˜ D
Revised: ∈ R N × n and define ˜ D k 1 ∈ R d × d and ˜ D

Original: k 2 ∈ R N − d × n − d
Revised: k 2 ∈ R N − d × n − d

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: ˜ D k(cid:21) .
Revised: With these definitions in place we can write [Equation] I [Equation]

Revised: F

Original: [Equation]
Revised: [Equation]

Original:  The first term in (22) is bounded in the regime n ≥ N , for some constant c and with probability at least 1 − 2 e − n/ 2 by
Revised: I [Equation] F . The first term in (23) is bounded in the regime n ≥ N , for some constant c and with probability at least 1 − 2 e − n/ 2 by

Original: [Equation]
Revised: [Equation]

Original: See Section G.1.1 for a proof.
Revised: See Section G.1.1 for a proof.

Original: The second term in (22) is bounded for some constant c , with probability at least 1 − 2 n − 10 − 2 e − d − e − n and in regime ( d + nσ 2 z ) log( n ) ≤ N by
Revised: The second term in (23) is bounded for some constant c , with probability at least 1 − 2 n − 10 − 2 e − d − e − n and in regime ( d + nσ 2 z ) log( n ) ≤ N by

Original: [Equation]
Revised: [Equation]

Original: See Section G.1.2 for a proof.
Revised: See Section G.1.2 for a proof.

Original: The third term in (22) is bounded for some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N and in the regime ( d + nσ 2 z ) log( n ) ≤ N ,
Revised: The third term in (23) is bounded for some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N and in the regime ( d + nσ 2 z ) log( n ) ≤ N ,

Original: N log( N ) ≤ n and 9 d ≤ N by
Revised: N log( N ) ≤ n and 9 d ≤ N by

Original: [Equation]
Revised: [Equation]

Original: See Section G.1.3 for a proof.
Revised: See Section G.1.3 for a proof

Original: Combining those results we can bound equation (22) in the regime N log( N ) ≤ n , 9 d ≤ N and ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − 2 e − N/ 18 − 3 n − 10 −

Original: [Equation]
Revised: [Equation]

Original:  e [Equation] N − 2 e − n/ 2 as
Revised: Combining those results we can bound equation (23) in the regime N log( N ) ≤ n , 9 d ≤ N and ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N − 2 e − n/ 2 as

Original: [Equation]
Revised: [Equation]

Original: This concludes the proof of equation (8).
Revised: This concludes the proof of equation (9) [Equation]

Original: G.1. P ROOF OF EQUATION (23)
Revised: G.1. P ROOF OF EQUATION (24)

Original: In the regime n ≥ N , for some constant c and with probability at least 1 − 2 e − n/ 2 , the first term in (22) is bounded by
Revised: In the regime n ≥ N , for some constant c and with probability at least 1 − 2 e − n/ 2 , the first term in (23) is bounded by

Original: [Equation]
Revised: [Equation]

Original: Inequality (i) uses (cid:13)(cid:13)
Revised: [Table] (ii [Table]

Original: U Ty
Revised: D [Table] [Table] [Table]

Original: [Equation]
Revised: [Table]

Original: U
Revised: i

Original: [Equation]
Revised: [Table] [Table]

Original: ≤ 1 .
Revised: σ y

Original: To obtain inequality (ii), we used that ˜ D k 1 = diag ( . . . , (1 − (1 − ησ 2 y,i ) k ) /σ
Revised: [Table]

Original: y,i , . . . )
Revised: i [Table]

Original: ∈ R d × d and that
Revised: [Table] R d [Table]

Original: [Equation]
Revised: [Equation]

Original: Here, the last inequality holds in the regime n ≥ N , for some constant c and with probability at least 1 − 2 e − n/ 2 and follows from Section H.4, equation (63).

Original: G.1. P ROOF OF EQUATION (24)
Revised: G.1. P ROOF OF EQUATION (25)

Original: For some constant c , with probability at least 1 − 2 n − 10 − 4 e − d − e − n and in regime ( d + nσ 2 z ) log( n ) ≤ N the second term in (22) is bounded by
Revised: For some constant c , with probability at least 1 − 2 n − 10 − 4 e − d − e − n and in regime ( d + nσ 2 z ) log( n ) ≤ N the second term in (23) is bounded by

Original: [Equation]
Revised: [Equation]

Original: Here, inequality (i) follows from (cid:13)(cid:13) V Tx 1 V y 1 (cid:13)(cid:13) ≤ 1 and (cid:13)(cid:13)
Revised: Here, inequality (i) follows from (cid:13)(cid:13) V Tx 1 V y 1 (cid:13)(cid:13) ≤ 1 and (cid:13)(cid:13)

Original: U Ty 1 U (cid:13)(cid:13) ≤ 1 .
Revised: U Ty 1 U (cid:13)(cid:13) ≤ 1 .

Original: For inequality (ii) we used that ˜ D
Revised: For inequality (ii) we used that ˜ D

Original: k 1 = diag (cid:0) . . .
Revised: k 1 = diag (cid:0) . . .

Original: , (1 − (1 − ησ 2 y,i ) k ) /σ
Revised: , (1 − (1 − ησ 2 y,i ) k ) /σ

Original: y,i , . . .
Revised: y,i , . . .

Original: (cid:1)
Revised: (cid:1)

Original: ∈ R d × d and the bound on (cid:13)(cid:13) U
Revised: ∈ R d × d and the bound on (cid:13)(cid:13) U

Original: Ty 2 U x 1 (cid:13)(cid:13) from
Revised: Ty 2 U x 1 (cid:13)(cid:13) from

Original: Section H.1, equation (34) that holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − 2 n − 10 − 2 e − d − e − n .
Revised: Section H.1, equation (35) that holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , for some constant c and with probability at least 1 − 2 n − 10 − 2 e − d − e − n .

Original: G.1. P ROOF OF EQUATION (25)
Revised: G.1. P ROOF OF EQUATION (26)

Original: For some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N and in the regime ( d + nσ 2 z ) log( n ) ≤ N and 9 d ≤ N we obtain
Revised: For some constant c , with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N and in the regime ( d + nσ 2 z ) log( n ) ≤ N and 9 d ≤ N we obtain

Original: [Equation]
Revised: [Equation]

Original: Here, inequality (i) follows from (cid:13)(cid:13) U Ty 1 U x 1 (cid:13)(cid:13) ≤ 1 and the bound in Section H.1, equations (33), (34) and (35) and  holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , N log( N ) ≤ n and 9 d ≤ N , for some constant c and with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N .
Revised: Here, inequality (i) follows from (cid:13)(cid:13) U Ty 1 U x 1 (cid:13)(cid:13) ≤ 1 and the bound in Section H.1, equations (34), (35) and (36) and holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , N log( N ) ≤ n and 9 d ≤ N , for some constant c and with probability at least 1 − 2 e − N/ 18 − 3 n − 10 − 3 e − d − e − n − e − N .

Original: Inequality (ii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , which implies γ < 

Revised: Inequality (ii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N , which implies γ <

Original: and we used the definition of ˜ D kfrom equation (21).
Revised: and we used the definition of ˜ D kfrom equation (22).

Original: G. P ROOF OF EQUATION (9)
Revised: G. P ROOF OF EQUATION (10)

Original: Recall that D k = diag (cid:0) . . .
Revised: Recall that D k = diag (cid:0) . . .

Original: , (1 − (1 − ησ 2 y,i ) k ) /σ
Revised: , (1 − (1 − ησ 2 y,i ) k ) /σ

Original: y,i , . . .
Revised: y,i , . . .

Original: (cid:1) ∈ R N × N .
Revised: (cid:1) ∈ R N × N .

Original: The second term in equation (5) can be bounded as
Revised: The second term in equation (6) can be bounded as

Original: [Equation]
Revised: [Equation]

Original: where we used that (cid:13)(cid:13) V Tx V y (cid:13)(cid:13) 2 ≤ 1 and the definition of D k from equation (21).
Revised: where we used that (cid:13)(cid:13) V Tx V y (cid:13)(cid:13) 2 ≤ 1 and the definition of D k from equation (22).

Original: H A UXILIARY PROOFS
Revised: H A UXILIARY PROOFS

Original: In this Section we provide a summary of auxiliary proofs that are used to prove the main results in
Revised: In this Section we provide a summary of auxiliary proofs that are used to prove the main results in

Original: Sections F and G.
Revised: Sections F and G.

Original: H. A PPLYING THE SIN - THETA THEOREM TO BOUND THE DISTANCE BETWEEN SUBSPACES
Revised: H. A PPLYING THE SIN - THETA THEOREM TO BOUND THE DISTANCE BETWEEN SUBSPACES

Original: In this Section, we use the following variant (Cai et al., 2015, Prop. 1) of the sin-theta theorem (Davis & Kahan, 1970) to bound the distances between subspaces occurring in the proofs in Section F and G.
Revised: In this Section, we use the following variant (Cai et al., 2015, Prop. 1) of the sin-theta theorem (Davis & Kahan, 1970) to bound the distances between subspaces occurring in the proofs in Section F and G.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let Q and ˆ Q be n × n symmetric matrices.
Revised: Let Q and ˆ Q be n × n symmetric matrices.

Original: Let r < n be arbitrary and let U and ˆ U be formed by the r leading singular vectors of Q and ˆ Q .
Revised: Let r < n be arbitrary and let U and ˆ U be formed by the r leading singular vectors of Q and ˆ Q .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Recall that Y =
Revised: Recall that Y =

Original: U y Σ
Revised: U y Σ

Original: y V Ty ∈ R n × N and X
Revised: y V Ty ∈ R n × N and X

Original: =
Revised: =

Original: U x Σ x V Tx ∈ R n × N are the SVDs of Y and
Revised: U x Σ x V Tx ∈ R n × N are the SVDs of Y and

Original: X respectively.
Revised: X respectively.

Original: Let U
Revised: Let U

Original: y 1 , U x 1 ∈ R n × d be the d leading left singular vectors of Y and X , let
Revised: y 1 , U x 1 ∈ R n × d be the d leading left singular vectors of Y and X , let

Original: U y 2 , U x 2 ∈ R n × n − d be the orthonormal complements.
Revised: U y 2 , U x 2 ∈ R n × n − d be the orthonormal complements.

Original: Let ˜ U
Revised: Let ˜ U

Original: y
Revised: y

Original: =
Revised: =

Original: [ U y
Revised: [ U y

Original: U y 2 ] ∈ R n × n and ˜ U x
Revised: U y 2 ] ∈ R n × n and ˜ U x

Original: = [ U x
Revised: = [ U x

Original: U x 2 ] ∈ R n × n .
Revised: U x 2 ] ∈ R n × n .

Original: Analogous definitions can be made for the leading right singular vectors of Y and X .
Revised: Analogous definitions can be made for the leading right singular vectors of Y and X .

Original: We start by applying Proposition 1 to bound the distance between the subspaces spanned by the d leading left singular vectors U y 1 of Y and the subspace model U as
Revised: We start by applying Proposition 1 to bound the distance between the subspaces spanned by the d leading left singular vectors U y 1 of Y and the subspace model U as

Original: [Equation]
Revised: [Equation]

Original: where inequality (i) follows from Proposition 1 and inequality (ii) holds with probability at least 1 − n − 10 − 2 e − d + e − n and follows from Section H.2 equation (44).
Revised: where inequality (i) follows from Proposition 1 and inequality (ii) holds with probability at least 1 − n − 10 − 2 e − d + e − n and follows from Section H.2 equation (45).

Original: Inequality (iii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N .
Revised: Inequality (iii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N .

Original: Next, we establish a bound for the distance between the subspaces spanned by the d leading left singular vectors of X and Y .
Revised: Next [Equation] we establish a bound for the distance between the subspaces spanned by the d leading left singular vectors of X and Y .

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original:  where inequality (i) follows from Proposition 1 and inequality (ii) holds for some constant c and with probability
Revised: N [Equation] where inequality (i) follows from Proposition 1 and inequality (ii) holds for some constant c and with probability

Original: at least 1 − 2 n − 10 − 2 e − d + e − n and follows from the results in Section H.2 equations (44),(45).
Revised: at least 1 − 2 n − 10 − 2 e − d + e − n and follows from the results in Section H.2 equations (45),(46).

Original: Inequality (iii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N .
Revised: Inequality (iii) holds in the regime ( d + nσ 2 z ) log( n ) ≤ N .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Inequality (i) follows from Proposition 1.
Revised: Inequality (i) follows from Proposition 1.

Original: Inequality (ii) follows from the fact that σ 2 x,d +1 = 0 and that 1 c N ≤ σ 2 x,d holds in the regime 9 d ≤ N , with probability at least 1 − 2 e − N/ 18 and for some constant c (see Section H.4 equation (62)).
Revised: Inequality (ii) follows from the fact that σ 2 x,d +1 = 0 and that 1 c N ≤ σ 2 x,d holds in the regime 9 d ≤ N , with probability at least 1 − 2 e − N/ 18 and for some constant c (see Section H.4 equation (63)).

Original: Inequality (iii) holds for some constant c and with probability at least 1 − 2 n − 10 − e − n − e − N − e − d and follows from the results in Section H.2 equations (46), (37).
Revised: Inequality (iii) holds for some constant c and with probability at least 1 − 2 n − 10 − e − n − e − N − e − d and follows from the results in Section H.2 equations (47), (38).

Original: Inequality (iv) holds in the regime
Revised: Inequality (iv) holds in the regime

Original: N log( N ) ≤ n .
Revised: N log( N ) ≤ n .

Original: Inequality (v) holds in the regime nσ 2 z log( n ) ≤ N .
Revised: Inequality (v) holds in the regime nσ 2 z log( n ) ≤ N .

Original: To abbreviate notation from now on we define
Revised: To abbreviate notation from now on we define

Original: [Equation]
Revised: [Equation]

Original: H. B OUNDING A SUM OF INDEPENDENT RANDOM MATRICES
Revised: H. B OUNDING A SUM OF INDEPENDENT RANDOM MATRICES

Original: The Matrix Bernstein inequality (Oliveira, 2010; Tropp, 2012) can be used to bound a sum of independent, bounded and centered random matrices.
Revised: The Matrix Bernstein inequality (Oliveira, 2010; Tropp, 2012) can be used to bound a sum of independent, bounded and centered random matrices.

Original: We state the theorem below and then show how we applied it to bound several terms occurring in Sections F and G.
Revised: We state the theorem below and then show how we applied it to bound several terms occurring in Sections F and G.

Original: Theorem 3 (Matrix Bernstein) .
Revised: Theorem 3 (Matrix Bernstein) .

Original: Let S 1 , . .
Revised: Let S 1 , . .

Original: , S n be independent, centered random matrices with common dimension d × d , and assume that each one is uniformly bounded
Revised: , S n be independent, centered random matrices with common dimension d × d , and assume that each one is uniformly bounded

Original: E [ S k ] = 0 and ∥ S k ∥ ≤ L for each k = 1 , . .
Revised: E [ S k ] = 0 and ∥ S k ∥ ≤ L for each k = 1 , . .

Original: n.
Revised: n.

Original: Introduce the sum
Revised: Introduce the sum

Original: [Equation]
Revised: [Equation]

Original: and let v ( Z ) denote the matrix variance statistics of the sum:
Revised: and let v ( Z ) denote the matrix variance statistics of the sum:

Original: [Equation]
Revised: [Equation]

Original: Then with probability at least 1 − e − δ and δ ≥
Revised: Then with probability at least 1 − e − δ and δ ≥

Original: [Equation]
Revised: [Equation]

Original: Recall our signal model to be Y =
Revised: Recall our signal model to be Y =

Original: X
Revised: X

Original: + Z =
Revised: + Z =

Original: UC + Z , with entries
Revised: UC + Z , with entries

Original: i.i.d.
Revised: i.i.d.

Original: as c i,j ∼ N (0 , 1) and z i,j ∼ N (0 , σ 2 z ) and dimensions Y , X , Z ∈ R n × N , C ∈ R d × N .
Revised: as c i,j ∼ N (0 , 1) and z i,j ∼ N (0 , σ 2 z ) and dimensions Y , X , Z ∈ R n × N , C ∈ R d × N .

Original: The subspace matrix U ∈ R n × d has orthonormal columns.
Revised: The subspace matrix U ∈ R n × d has orthonormal columns.

Original: We start by applying Theorem 3 to establish the following bound.
Revised: We start by applying Theorem 3 to establish the following bound.

Original: With probability 1 − N − 10 − e − N − e − d
Revised: With probability 1 − N − 10 − e − N − e − d

Original: and for some unspecified numerical constant c we have
Revised: and for some unspecified numerical constant c we have

Original: [Equation]
Revised: [Equation]

Original: Proof of equation (37) : We define ˜ Z
Revised: Proof of equation (38) : We define ˜ Z

Original: =
Revised: =

Original: U
Revised: U

Original: T Z ∈ R d × N .
Revised: T Z ∈ R d × N .

Original: Note that the entries in ˜ Z are independent and identically distributed like the entries in Z
Revised: Note that the entries in ˜ Z are independent and identically distributed like the entries in Z

Original: , i.e., ˜ z i,j ∼ N (0 , σ 2 z ) .
Revised: , i.e., ˜ z i,j ∼ N (0 , σ 2 z ) .

Original: Next we check the conditions of applying Theorem 3 to bound (cid:13)(cid:13)(cid:13) C T ˜ Z (cid:13)(cid:13)(cid:13) .
Revised: Next we check the conditions of applying Theorem 3 to bound (cid:13)(cid:13)(cid:13) C T ˜ Z (cid:13)(cid:13)(cid:13) .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: with c i , ˜ z
Revised: with c i , ˜ z

Original: [ i ] ∈ R N being the rows of C , ˜ Z .
Revised: [ i ] ∈ R N being the rows of C , ˜ Z .

Original: Since c i has zero mean,
Revised: Since c i has zero mean,

Original: [Equation]
Revised: [Equation]

Original: , for all i = 1 , . . .
Revised: , for all i = 1 , . . .

Original: , d .
Revised: , d .

Original: Further, we have
Revised: Further, we have

Original: [Equation]
Revised: [Equation]

Original: 39) where the inequality follows from equations (56),(55) and holds with probability at least 1 − e − N − e − d .
Revised: 40) where the inequality follows from equations (57),(56) and holds with probability at least 1 − e − N − e − d .

Original: Finally we need to compute the matrix variance statistic v ( C T ˜ Z ) .
Revised: Finally we need to compute the matrix variance statistic v ( C T ˜ Z ) .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: With equations (39),(41) in place we are ready to apply Theorem 3 to obtain with probability at least 1 − N − 10 − e − N − e − d
Revised: With equations (40),(42) in place we are ready to apply Theorem 3 to obtain with probability at least 1 − N − 10 − e − N − e − d

Original: and some constant c
Revised: and some constant c

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since d < N .
Revised: where the last inequality holds since d < N .

Original: This concludes the proof of equation (37).
Revised: This concludes the proof of equation (38).

Original: In the remainder of this Section we apply the example in Tropp (2015, Sec. 1.6.3) that illustrates how to use Theorem 3 to bound the distance between sample and true covariance matrices.
Revised: In the remainder of this Section we apply the example in Tropp (2015, Sec. 1.6.3) that illustrates how to use Theorem 3 to bound the distance between sample and true covariance matrices.

Original: With probability at least 1 − n − 10 and an unspecified numerical constant c we have
Revised: With probability at least 1 − n − 10 and an unspecified numerical constant c we have

Original: [Equation]
Revised: [Equation]

Original: where we assume that the l 2 norm of the random vector a ∈ R n is bounded ∥ a ∥ 22 ≤ B .
Revised: where we assume that the l 2 norm of the random vector a ∈ R n is bounded ∥ a ∥ 22 ≤ B .

Original: In the following we show how to apply (43) to establish the following bounds.
Revised: In the following we show how to apply (44) to establish the following bounds.

Original: With probability at least 1 − n − 10 − 2 e − d + e − n and for some unspecified numerical constant c
Revised: With probability at least 1 − n − 10 − 2 e − d + e − n and for some unspecified numerical constant c

Original: [Equation]
Revised: [Equation]

Original: With probability at least 1 − n − 10 − e − d and for some unspecified numerical constant c
Revised: With probability at least 1 − n − 10 − e − d and for some unspecified numerical constant c

Original: [Equation]
Revised: [Equation]

Original: With probability at least 1 − n − 10 − e − n and for some unspecified numerical constant c
Revised: With probability at least 1 − n − 10 − e − n and for some unspecified numerical constant c

Original: [Equation]
Revised: [Equation]

Original: In the notation of the general example in (43), the proofs of (44)-(46) consist of deriving expressions for ∥ a ∥ 22 ≤ B and (cid:13)(cid:13) E (cid:2) aa T (cid:3)(cid:13)(cid:13) respectively.
Revised: In the notation of the general example in (44), the proofs of (45)-(47) consist of deriving expressions for ∥ a ∥ 22 ≤ B and (cid:13)(cid:13) E (cid:2) aa T (cid:3)(cid:13)(cid:13) respectively.

Original: Proof of equation (44) :
Revised: Proof of equation (45) :

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: for some constant c and where the first inequality holds with probability at least 1 − 2 e − d − e − n as it is shown in Section H.3.
Revised: for some constant c and where the first inequality holds with probability at least 1 − 2 e − d − e − n as it is shown in Section H.3.

Original: Further, we have
Revised: Further, we have

Original: [Equation]
Revised: [Equation]

Original: Since practical noise levels satisfy σ 2 z ≤ 1 , there exists some constant c such that
Revised: Since practical noise levels satisfy σ 2 z ≤ 1 , there exists some constant c such that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 e − d − e − n .
Revised: with probability at least 1 − 2 e − d − e − n .

Original: Inserting (47) and (49) in the general form in (43) concludes the proof of equation (44).
Revised: Inserting (48) and (50) in the general form in (44) concludes the proof of equation (45).

Original: Proof of equation (45) :
Revised: Proof of equation (46) :

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: for some constant c and where the inequality holds with probability at least 1 − e − d as it is shown in
Revised: for some constant c and where the inequality holds with probability at least 1 − e − d as it is shown in

Original: Section H.3.
Revised: Section H.3.

Original: Further, we have
Revised: Further, we have

Original: [Equation]
Revised: [Equation]

Original: Inserting(50) and (51)  in the general form in (43) concludes the proof of equation (45).
Revised: Inserting (51) and (52) in the general form in (44) concludes the proof of equation (46).

Original: Proof of equation (46) :
Revised: Proof of equation (47) :

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: for some constant c and where the inequality holds with probability at least 1 − e − n as it is shown in
Revised: for some constant c and where the inequality holds with probability at least 1 − e − n as it is shown in

Original: Section H.3.
Revised: Section H.3.

Original: Further, we have
Revised: Further, we have

Original: [Equation]
Revised: [Equation]

Original: Inserting(52) and (53)  in the general form in (43) concludes the proof of equation (46).
Revised: Inserting (53) and (54) in the general form in (44) concludes the proof of equation (47).

Original: H. T AIL BOUNDS FOR INNER PRODUCTS OF G AUSSIAN VECTORS
Revised: H. T AIL BOUNDS FOR INNER PRODUCTS OF G AUSSIAN VECTORS

Original: Recall that c ∼ N (0 , I ) ∈
Revised: Recall that c ∼ N (0 , I ) ∈

Original: R d and z ∼ N (0 , σ 2 z I ) ∈ R n are the columns of C , Z respectively.
Revised: R d and z ∼ N (0 , σ 2 z I ) ∈ R n are the columns of C , Z respectively.

Original: Also
Revised: Also

Original: U ∈ R n × d has orthonormal columns.
Revised: U ∈ R n × d has orthonormal columns.

Original: In this section we state some relations on the concentration of the inner products between those vectors.
Revised: In this section we state some relations on the concentration of the inner products between those vectors.

Original: The chi-squared distributed c T c can be bounded with probability at most e − t as
Revised: The chi-squared distributed c T c can be bounded with probability at most e − t as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Substituting t with the signal dimension d gives
Revised: Substituting t with the signal dimension d gives

Original: [Equation]
Revised: [Equation]

Original: with probability at most e − d .
Revised: with probability at most e − d .

Original: Using the same result we can write
Revised: Using the same result we can write

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with probability at most e − n and where z ′ ∼ N (0 , I ) .
Revised: with probability at most e − n and where z ′ ∼ N (0 , I ) .

Original: Next we show that with probability at most 2 e − d
Revised: Next we show that with probability at most 2 e − d

Original: [Equation]
Revised: [Equation]

Original: To this end, note that for any (deterministic) vector c we have z T
Revised: To this end, note that for any (deterministic) vector c we have z T

Original: Uc ∼ N (0 , ∥ c ∥ 22 ) , and we apply a simple tail bound for Gaussian random variables to get
Revised: Uc ∼ N (0 , ∥ c ∥ 22 ) , and we apply a simple tail bound for Gaussian random variables to get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with probability at most e − t 2 .
Revised: with probability at most e − t 2 .

Original: It is straightforward to see that substituting t with √ d , applying (55) to bound ∥ c
Revised: It is straightforward to see that substituting t with √ d , applying (56) to bound ∥ c

Original: ∥ 2 and combining the results with a union bound results into the bound (57).
Revised: ∥ 2 and combining the results with a union bound results into the bound (58).

Original: We can combine the results from this section to bound the sum
Revised: We can combine the results from this section to bound the sum

Original: [Equation]
Revised: [Equation]

Original: with probability at most 2 e − d + e − n .
Revised: with probability at most 2 e − d + e − n .

Original: H.
Revised: H.

Original: B OUNDING THE EXTREME SINGULAR VALUES OF G AUSSIAN RANDOM MATRICES AND EMPIRICAL COVARIANCE MATRICES
Revised: B OUNDING THE EXTREME SINGULAR VALUES OF G AUSSIAN RANDOM MATRICES AND EMPIRICAL COVARIANCE MATRICES

Original: In this Section we state results for the extreme singular values of some of the matrices occurring in
Revised: In this Section we state results for the extreme singular values of some of the matrices occurring in

Original: Sections F and G. Specifically, we establish equations (16),(17),(28) and (35) (ii).
Revised: Sections F and G. Specifically, we establish equations (17),(18),(29) and (36) (ii).

Original: A standard deviation inequality for the extreme singular values of some matrix A ∈ R M × m with independent and identically standard normal distributed entries implies that (Rudelson & Vershynin, 2010, equation (2.3))
Revised: A standard deviation inequality for the extreme singular values of some matrix A ∈ R M × m with independent and identically standard normal distributed entries implies that (Rudelson & Vershynin, 2010, equation (2.3))

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 e − t 2 / 2 for t ≥ 0 .
Revised: with probability at least 1 − 2 e − t 2 / 2 for t ≥ 0 .

Original: We start with the largest singular value σ x,max of the feature matrix X ∈ R n × N .
Revised: We start with the largest singular value σ x,max of the feature matrix X ∈ R n × N .

Original: Recall that we have
Revised: Recall that we have

Original: X = UC with orthonormal U ∈ R n × d and C ∈ R d × N with i.i.d.
Revised: X = UC with orthonormal U ∈ R n × d and C ∈ R d × N with i.i.d.

Original: entries c i,j ∼ N (0 , 1) .
Revised: entries c i,j ∼ N (0 , 1) .

Original: Hence, the singular values of UC are the singular values of C .
Revised: Hence, the singular values of UC are the singular values of C .

Original: Applying equation (60) to the matrix C under the assumption that the hidden signal dimension d and the number of training examples N fulfill 4 d ≤ N and choosing t = √ N/ 2 we obtain[Equation]
Revised: Applying equation (61) to the matrix C under the assumption that the hidden signal dimension d and the number of training examples N fulfill 4 d ≤ N and choosing t = √ N/ 2 we obtain

Original: [Equation]
Revised: [Equation]

Original: which holds with probability at least 1 − 2 e − N/ 8 .
Revised: which holds with probability at least 1 − 2 e − N/ 8 .

Original: This establishes equation (16).
Revised: This establishes equation (17).

Original: For σ x,min the smallest non-zero singular value of X we have σ x,min = σ x,d = σ c,d .
Revised: For σ x,min the smallest non-zero singular value of X we have σ x,min = σ x,d = σ c,d .

Original: In the regime 9 d ≤ N and choosing t = √ N/ 3 we obtain
Revised: In the regime 9 d ≤ N and choosing t = √ N/ 3 we obtain

Original: [Equation]
Revised: [Equation]

Original: which holds with probability at least 1 − 2 e − N/ 18 and for some constant c .
Revised: which holds with probability at least 1 − 2 e − N/ 18 and for some constant c .

Original: This establishes (35) (ii).
Revised: This establishes (36) (ii).

Original: Next, we state a bound on the largest singular value ∥
Revised: Next, we state a bound on the largest singular value ∥

Original: Z ∥ of the noise matrix Z ∈ R n × N with i.i.d.
Revised: Z ∥ of the noise matrix Z ∈ R n × N with i.i.d.

Original: entries z
Revised: entries z

Original: i,j ∼ N (0 , σ 2 z ) .
Revised: i,j ∼ N (0 , σ 2 z ) .

Original: Note that Z = σ z ˜ Z , where the entries of ˜ Z follow ˜ z
Revised: Note that Z = σ z ˜ Z , where the entries of ˜ Z follow ˜ z

Original: i,j ∼ N (0 , 1) .
Revised: i,j ∼ N (0 , 1) .

Original: Applying equation (60) with t = √ N + √ n yields 
Revised: Applying equation (61) with t = √ N + √ n yields [Equation]

Original: [Equation]
Revised: [Equation]

Original: 63) with probability at least 1 − 2 e − n/ 2 and some constant c .
Revised: with probability at least 1 − 2 e − n/ 2 and some constant c .

Original: For the last inequality we assumed n ≥ N .
Revised: For the last inequality we assumed n ≥ N .

Original: This establishes equation (28).
Revised: This establishes equation (29).

Original: Finally, we derive bounds for some of the squared singular values of Y ∈ R n × N .
Revised: Finally, we derive bounds for some of the squared singular values of Y ∈ R n × N .

Original: In particular, we show that
Revised: In particular, we show that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: and finally
Revised: and finally

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: both hold with probability at least 1 − e − d − e − n − n − 9 and for some constants C and ϵ ∈ (0 , 1) in the regime N
Revised: both hold with probability at least 1 − e − d − e − n − n − 9 and for some constants C and ϵ ∈ (0 , 1) in the regime N

Original: [Equation]
Revised: [Equation]

Original: This establishes equation (17).
Revised: This establishes equation (18).

Original: To this end, we rely on the following Corollary (Vershynin, 2011, Corollary 5.52).
Revised: To this end, we rely on the following Corollary (Vershynin, 2011, Corollary 5.52).

Original: Corollary 1 (Covariance estimation for arbitrary distributions) .
Revised: Corollary 1 (Covariance estimation for arbitrary distributions) .

Original: Let x be a random vector in R n supported in some centered Euclidean ball whose radius we denote √ m .
Revised: Let x be a random vector in R n supported in some centered Euclidean ball whose radius we denote √ m .

Original: Consider N independent samples x i arranged as columns of the random matrix A ∈ R n × N .
Revised: Consider N independent samples x i arranged as columns of the random matrix A ∈ R n × N .

Original: Denote Σ N = 1 N AA T as the sample covariance matrix and Σ as the true covariance matrix.
Revised: Denote Σ N = 1 N AA T as the sample covariance matrix and Σ as the true covariance matrix.

Original: Let ϵ ∈
Revised: Let ϵ ∈

Original: (0 , 1) and t ≥ 1 .
Revised: (0 , 1) and t ≥ 1 .

Original: Then the following holds with probability at least 1 − n − t 2 :
Revised: Then the following holds with probability at least 1 − n − t 2 :

Original: [Equation]
Revised: [Equation]

Original: Here C is an absolute constant.
Revised: Here C is an absolute constant.

Original: We apply Corollary 1 to the random vectors y =
Revised: We apply Corollary 1 to the random vectors y =

Original: Uc + z .
Revised: Uc + z .

Original: Note that E (cid:2) yy T (cid:3) =
Revised: Note that E (cid:2) yy T (cid:3) =

Original: UU T + σ 2 z
Revised: UU T + σ 2 z

Original: I .
Revised: I .

Original: Further note that E (cid:104) ∥
Revised: Further note that E (cid:104) ∥

Original: y ∥ 22 (cid:105) =
Revised: y ∥ 22 (cid:105) =

Original: d + σ 2 z n .
Revised: d + σ 2 z n .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from Section H.3, (55), (56) and holds with probability at least 1 − e − d − e − n .
Revised: where the second inequality follows from Section H.3, (56), (57) and holds with probability at least 1 − e − d − e − n .

Original: Now we can apply Corollary 1 to make the following statement.
Revised: Now we can apply Corollary 1 to make the following statement.

Original: For some constant C and ϵ ∈ (0 , 1) if N ≥ Ctϵ − 2 ( d + σ 2 z n ) log n , then with probability at least 1 − e − d − e − n − n − t
Revised: For some constant C and ϵ ∈ (0 , 1) if N ≥ Ctϵ − 2 ( d + σ 2 z n ) log n , then with probability at least 1 − e − d − e − n − n − t

Original: [Equation]
Revised: [Equation]

Original: Consequently the singular values σ i ( 1 N YY T ) and σ i ( E (cid:2)
Revised: Consequently the singular values σ i ( 1 N YY T ) and σ i ( E (cid:2)

Original: yy T (cid:3) ) differ by at most ϵ (1 + σ 2 z ) and we can bound
Revised: yy T (cid:3) ) differ by at most ϵ (1 + σ 2 z ) and we can bound

Original: [Equation]
Revised: [Equation]

Original: 69) and further
Revised: and further

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of equations (64) and (65).
Revised: which concludes the proof of equations (65) and (66).

Original: In the same manner we bound
Revised: In the same manner we bound

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of (66)
Revised: which concludes the proof of (67)

Original: [Equation]
Revised: [Equation]