Original: A SYNC -RED: A P ROVABLY C ONVERGENT A SYNCHRONOUS B LOCK P ARALLEL S TOCHASTIC M ETHOD USING D EEP D ENOISING P RIORS
Revised: A SYNC -RED: A P ROVABLY C ONVERGENT A SYNCHRONOUS B LOCK P ARALLEL S TOCHASTIC M ETHOD USING D EEP D ENOISING P RIORS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Regularization by denoising (RED) is a recently developed framework for solving inverse problems by integrating advanced denoisers as image priors.
Revised: Regularization by denoising (RED) is a recently developed framework for solving inverse problems by integrating advanced denoisers as image priors.

Original: Recent work has shown its state-of-the-art performance when combined with pre-trained deep denoisers .
Revised: Recent work has shown its state-of-the-art performance when combined with pre-trained deep denoisers .

Original: However, current RED algorithms are inadequate for parallel processing on multicore systems.
Revised: However, current RED algorithms are inadequate for parallel processing on multicore systems.

Original: We address this issue by proposing a new asynchronous
Revised: We address this issue by proposing a new asynchronous

Original: RED (A SYNC -RED) algorithm that enables asynchronous parallel processing of data, making it signiﬁcantly faster than its serial counterparts for large-scale inverse problems.
Revised: RED (A SYNC -RED) algorithm that enables asynchronous parallel processing of data, making it signiﬁcantly faster than its serial counterparts for large-scale inverse problems.

Original: The computational complexity of A SYNC -RED is further reduced by using a random subset of measurements at every iteration.
Revised: The computational complexity of A SYNC -RED is further reduced by using a random subset of measurements at every iteration.

Original: We present complete theoretical analysis of the algorithm by establishing its convergence under explicit assumptions on the data-ﬁdelity and the denoiser.
Revised: We present complete theoretical analysis of the algorithm by establishing its convergence under explicit assumptions on the data-ﬁdelity and the denoiser.

Original: We validate A SYNC -RED on image recovery using pre-trained deep denoisers as priors.
Revised: We validate A SYNC -RED on image recovery using pre-trained deep denoisers as priors.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imaging inverse problems seek to recover an unknown image x 2 R n from its noisy measurements y 2 R m .
Revised: Imaging inverse problems seek to recover an unknown image x 2 R n from its noisy measurements y 2 R m .

Original: Such problems arise in many ﬁelds, ranging from low-level computer vision to biomedical imaging.
Revised: Such problems arise in many ﬁelds, ranging from low-level computer vision to biomedical imaging.

Original: Since many imaging inverse problems are ill-posed, it is common to regularize the solution by using prior information on the unknown image.
Revised: Since many imaging inverse problems are ill-posed, it is common to regularize the solution by using prior information on the unknown image.

Original: Widely-adopted image priors include total variation, low-rank penalties, and transform-domain sparsity (Rudin et al., 1992; Figueiredo &
Revised: Widely-adopted image priors include total variation, low-rank penalties, and transform-domain sparsity (Rudin et al., 1992; Figueiredo &

Original: Nowak, 2001; 2003; Hu et al., 2012; Elad & Aharon, 2006).
Revised: Nowak, 2001; 2003; Hu et al., 2012; Elad & Aharon, 2006).

Original: There has been considerable recent interest in plug-and-play priors (PnP) (Venkatakrishnan et al., 2013; Sreehari et al., 2016) and regularization by denoising (RED) (Romano et al., 2017), as frameworks for exploiting image denoisers as priors for image recovery.
Revised: There has been considerable recent interest in plug-and-play priors (PnP) (Venkatakrishnan et al., 2013; Sreehari et al., 2016) and regularization by denoising (RED) (Romano et al., 2017), as frameworks for exploiting image denoisers as priors for image recovery.

Original: The popularity of deep learning has led to a wide adoption of deep denoisers within PnP/RED, leading to their state-of-the-art performance in a variety of applications, including image restoration (Mataev et al., 2019), phase retrieval (Metzler et al., 2018), and tomographic imaging (Wu et al., 2020).
Revised: The popularity of deep learning has led to a wide adoption of deep denoisers within PnP/RED, leading to their state-of-the-art performance in a variety of applications, including image restoration (Mataev et al., 2019), phase retrieval (Metzler et al., 2018), and tomographic imaging (Wu et al., 2020).

Original: Their empirical success has also prompted a follow-up theoretical work clarifying the existence of explicit regularizers (Reehorst & Schniter, 2019), providing new interpretations based on ﬁxed-point projections (Cohen et al., 2020), and analyzing their coordinate/online variants (Sun et al., 2019a; Wu et al., 2020).
Revised: Their empirical success has also prompted a follow-up theoretical work clarifying the existence of explicit regularizers (Reehorst & Schniter, 2019), providing new interpretations based on ﬁxed-point projections (Cohen et al., 2020), and analyzing their coordinate/online variants (Sun et al., 2019a; Wu et al., 2020).

Original: Nonetheless, current PnP/RED algorithms are inherently serial , which makes them suboptimal for large-scale inverse problems on multicore systems (see Fig.
Revised: Nonetheless, current PnP/RED algorithms are inherently serial .

Original: 1 for an illustration).
Revised: As illustrated in Fig.

Revised: 1, this makes them suboptimal on multicore systems that are often required for processing large-scale datasets (Recht et al., 2011), such as those involving biomedical (Ong et al., 2020) and astronomical images (Akiyama et al., 2019)

Original: We address this gap by proposing a novel asynchronous RED (A SYNC -RED) algorithm.
Revised: We address this gap by proposing a novel asynchronous RED (A SYNC -RED) algorithm.

Original: The algorithm decomposes the inference problem into a sequence of partial (block-coordinate) updates on x executed asynchronously in parallel over a multicore system.
Revised: The algorithm decomposes the inference problem into a sequence of partial (block-coordinate) updates on x executed asynchronously in parallel over a multicore system.

Original: A SYNC -RED leads to a more efﬁcient usage of available cores by avoiding synchronization of partial updates.
Revised: A SYNC -RED leads to a more efﬁcient usage of available cores by avoiding synchronization of partial updates.

Original: A SYNC -RED is also scalable in terms of the number of measurements, since it processes only a small random subset of y at every iteration.
Revised: A SYNC -RED is also scalable in terms of the number of measurements, since it processes only a small random subset of y at every iteration.

Original: We present two new theoretical results on the convergence of A SYNC -RED based on a uniﬁed set of explicit assumptions on the data-ﬁdelity and the denoiser.
Revised: We present two new theoretical results on the convergence of A SYNC -RED based on a uniﬁed set of explicit assumptions on the data-ﬁdelity and the denoiser.

Original: Speciﬁcally, we establish its ﬁxed-point convergence in the batch setting and extend this analysis to the randomized minibatch scenario.
Revised: Speciﬁcally, we establish its ﬁxed-point convergence in the batch setting and extend this analysis to the randomized minibatch scenario.

Original: Our results extend recent work on serial block-coordinate RED  (Sun et al., 2019a) and are fully consistent with the traditional asynchronous parallel optimization methods (Lianet al., 2015;
Revised: Our results extend recent work on serial block-coordinate RED (BC-RED) (Sun et al., 2019a) and are fully consistent with the traditional asynchronous parallel optimization methods (Lian

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Core
Revised: Core

Original: [Table]
Revised: [Table]

Original: Core
Revised: Core

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +04cv1vrwy5ui3rQxe08u11sjxN8SI9ufzavGi93tkffbH/3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit> x<latexit sha1_base64="uVoSkjI+
Revised: +04cv1vrwy5ui3rQxe08u11sjxN8SI9ufzavGi93tkffbH/3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit> x<latexit sha1_base64="uVoSkjI+

Original: [Table]
Revised: [Table]

Original: +AhU7/EH/CyZdk=">AAAMMXicjZbdTtxGFMed9Cth
Revised: +AhU7/EH/CyZdk=">AAAMMXicjZbdTtxGFMed9Cth

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /OL86vzm/D74Y/Bu8Ofgrzr0/r0m5wundw3+/gdAo3rT</latexit> x<latexit sha1_base64="Pdx5Mwe0m4x5ntI0PCNaN3hWA4g=">AAAMMXicjZbdbtxEFMfd8tVm+WhBXHEzShSpSCHdLZUQF0hNNm0SKYqWNEmr4hKNvWN7iO1xZsbZ3Vh+B27hNXia3iFueQnO2J5jO7sgLCU+/p1z5uN/znjtZTFXejh8d+fue+9/8OFH9+6vDT7+5NPPHjz8/FyJXPrszBexkK89qljMU3amuY7Z60wymngxe+Vdjo3/1TWTiov0VC8y9jahYcoD7lMN6Nyde8nFtxcPNobbw+oiy8aoMTac5ppcPBx86U6Fnycs1X5MlfppNMz024JKzf2YlWturlhG/UsassIT4lJTT5WkvjZJJkXAlFkTjb+5ymnM9YJASMxUL5MmKhCp7mR6Mbg8QeWUJFRHRC0ST8RLWQbbJMi6pjIVOuJp2AtMuc8CSf12eF8k4NR2WBIISUaPn2wRpv3tXm7CfSmMoKXNtUSEkmbRoi9BUt6Ha5NwDZv1CQw+7QX4XLOyigilyDMCz1WB+jurRub+vDSBAQ9zeUswo4kWsPI6Ala/blokZVdX632N4lBIriNYdAmR+KhWR5W3BTasWq6OmJDsVh5L8wQ2lFSr0IJE9JqRmGnNJDE+JldsLsljzaWYtdWAzJClJpit6g5AeUwlqPG/wqPInJFbO5GLqYritFroVMACp8REKcJTQutRegnz6sSVa2ubm5tkIoUIYBKrgV7ErMhiytPSNLkxyA9NybfIlAU85WbfAKVIaLrmpmzWJBdwJjKhKn9ZTDoPvah2kLLYa+1eTHMvi9PG6HljliS0LI6qW8/jCyliEHRRFmM0exFw0PMkq2ffae3+KLBtmHtc3WqdRGaKIiRIBes3f2nhbrm2M92tsnZQGSYcXKaLRVbfVQBFCt0tcJRlJ47O
Revised: /OL86vzm/D74Y/Bu8Ofgrzr0/r0m5wundw3+/gdAo3rT</latexit> x<latexit sha1_base64="Pdx5Mwe0m4x5ntI0PCNaN3hWA4g=">AAAMMXicjZbdbtxEFMfd8tVm+WhBXHEzShSpSCHdLZUQF0hNNm0SKYqWNEmr4hKNvWN7iO1xZsbZ3Vh+B27hNXia3iFueQnO2J5jO7sgLCU+/p1z5uN/znjtZTFXejh8d+fue+9/8OFH9+6vDT7+5NPPHjz8/FyJXPrszBexkK89qljMU3amuY7Z60wymngxe+Vdjo3/1TWTiov0VC8y9jahYcoD7lMN6Nyde8nFtxcPNobbw+oiy8aoMTac5ppcPBx86U6Fnycs1X5MlfppNMz024JKzf2YlWturlhG/UsassIT4lJTT5WkvjZJJkXAlFkTjb+5ymnM9YJASMxUL5MmKhCp7mR6Mbg8QeWUJFRHRC0ST8RLWQbbJMi6pjIVOuJp2AtMuc8CSf12eF8k4NR2WBIISUaPn2wRpv3tXm7CfSmMoKXNtUSEkmbRoi9BUt6Ha5NwDZv1CQw+7QX4XLOyigilyDMCz1WB+jurRub+vDSBAQ9zeUswo4kWsPI6Ala/blokZVdX632N4lBIriNYdAmR+KhWR5W3BTasWq6OmJDsVh5L8wQ2lFSr0IJE9JqRmGnNJDE+JldsLsljzaWYtdWAzJClJpit6g5AeUwlqPG/wqPInJFbO5GLqYritFroVMACp8REKcJTQutRegnz6sSVa2ubm5tkIoUIYBKrgV7ErMhiytPSNLkxyA9NybfIlAU85WbfAKVIaLrmpmzWJBdwJjKhKn9ZTDoPvah2kLLYa+1eTHMvi9PG6HljliS0LI6qW8/jCyliEHRRFmM0exFw0PMkq2ffae3+KLBtmHtc3WqdRGaKIiRIBes3f2nhbrm2M92tsnZQGSYcXKaLRVbfVQBFCt0tcJRlJ47O

Original: [Table]
Revised: [Table]

Original: <
Revised: <

Original: [Table]
Revised: [Table]

Original: <
Revised: <

Original: [Table]
Revised: [Table]

Original: <
Revised: <

Original: [Table]
Revised: [Table]

Original: <
Revised: <

Original: [Table]
Revised: [Table]

Original: k
Revised: k

Original: [Table]
Revised: [Table]

Original: Core Core
Revised: Core Core

Original: Corey<latexit sha1_base64="+7+mD4Qola2MF8vLZ024UevUrMM
Revised: Corey<latexit sha1_base64="+7+mD4Qola2MF8vLZ024UevUrMM

Original: [Table]
Revised: [Table]

Original: Core Core
Revised: Core Core

Original: Corey<
Revised: Corey<

Original: [Table]
Revised: [Table]

Original: sha
Revised: sha

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: /WxeNi52tkdPt7//8enGs73mk/qB85Wz7jxxRs53zjPn0Jk4547v/OL86vzm/D74Y/Bu8Ofgrzr0/r0m5wundw3+/gdAo3rT</latexit> x<latexit sha1_base64="Pdx5Mwe0m4x5ntI0PCNaN3hWA4g=">AAAMMXicjZbdbtxEFMfd8tVm+WhBXHEzShSpSCHdLZUQF0hNNm0SKYqWNEmr4hKNvWN7iO1xZsbZ3Vh+B27hNXia3iFueQnO2J5jO7sgLCU+/p1z5uN/znjtZTFXejh8d+fue+9/8OFH9+6vDT7+5NPPHjz8/FyJXPrszBexkK89qljMU3amuY7Z60wymngxe+Vdjo3/1TWTiov0VC8y9jahYcoD7lMN6Nyde8nFtxcPNobbw+oiy8aoMTac5ppcPBx86U6Fnycs1X5MlfppNMz024JKzf2YlWturlhG/UsassIT4lJTT5WkvjZJJkXAlFkTjb+5ymnM9YJASMxUL5MmKhCp7mR6Mbg8QeWUJFRHRC0ST8RLWQbbJMi6pjIVOuJp2AtMuc8CSf12eF8k4NR2WBIISUaPn2wRpv3tXm7CfSmMoKXNtUSEkmbRoi9BUt6Ha5NwDZv1CQw+7QX4XLOyigilyDMCz1WB+jurRub+vDSBAQ9zeUswo4kWsPI6Ala/blokZVdX632N4lBIriNYdAmR+KhWR5W3BTasWq6OmJDsVh5L8wQ2lFSr0IJE9JqRmGnNJDE+JldsLsljzaWYtdWAzJClJpit6g5AeUwlqPG/wqPInJFbO5GLqYritFroVMACp8REKcJTQutRegnz6sSVa2ubm5tkIoUIYBKrgV7ErMhiytPSNLkxyA9NybfIlAU85WbfAKVIaLrmpmzWJBdwJjKhKn9ZTDoPvah2kLLYa+1eTHMvi9PG6HljliS0LI6qW8/jCyliEHRRFmM0exFw0PMkq2ffae3+KLBtmHtc3WqdRGaKIiRIBes3f2nhbrm2M92tsnZQGSYcXKaLRVbfVQBFCt0tcJRlJ47O
Revised: /WxeNi52tkdPt7//8enGs73mk/qB85Wz7jxxRs53zjPn0Jk4547v/OL86vzm/D74Y/Bu8Ofgrzr0/r0m5wundw3+/gdAo3rT</latexit> x<latexit sha1_base64="Pdx5Mwe0m4x5ntI0PCNaN3hWA4g=">AAAMMXicjZbdbtxEFMfd8tVm+WhBXHEzShSpSCHdLZUQF0hNNm0SKYqWNEmr4hKNvWN7iO1xZsbZ3Vh+B27hNXia3iFueQnO2J5jO7sgLCU+/p1z5uN/znjtZTFXejh8d+fue+9/8OFH9+6vDT7+5NPPHjz8/FyJXPrszBexkK89qljMU3amuY7Z60wymngxe+Vdjo3/1TWTiov0VC8y9jahYcoD7lMN6Nyde8nFtxcPNobbw+oiy8aoMTac5ppcPBx86U6Fnycs1X5MlfppNMz024JKzf2YlWturlhG/UsassIT4lJTT5WkvjZJJkXAlFkTjb+5ymnM9YJASMxUL5MmKhCp7mR6Mbg8QeWUJFRHRC0ST8RLWQbbJMi6pjIVOuJp2AtMuc8CSf12eF8k4NR2WBIISUaPn2wRpv3tXm7CfSmMoKXNtUSEkmbRoi9BUt6Ha5NwDZv1CQw+7QX4XLOyigilyDMCz1WB+jurRub+vDSBAQ9zeUswo4kWsPI6Ala/blokZVdX632N4lBIriNYdAmR+KhWR5W3BTasWq6OmJDsVh5L8wQ2lFSr0IJE9JqRmGnNJDE+JldsLsljzaWYtdWAzJClJpit6g5AeUwlqPG/wqPInJFbO5GLqYritFroVMACp8REKcJTQutRegnz6sSVa2ubm5tkIoUIYBKrgV7ErMhiytPSNLkxyA9NybfIlAU85WbfAKVIaLrmpmzWJBdwJjKhKn9ZTDoPvah2kLLYa+1eTHMvi9PG6HljliS0LI6qW8/jCyliEHRRFmM0exFw0PMkq2ffae3+KLBtmHtc3WqdRGaKIiRIBes3f2nhbrm2M92tsnZQGSYcXKaLRVbfVQBFCt0tcJRlJ47O

Original: [Table] [Table]
Revised: [Table] [Table]

Original: /3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit>
Revised: /3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit>

Original: [Table]
Revised: [Table]

Original: <latexit sha1_base64="uVoSkjI+4YofA+AhU7/EH/CyZdk=">AAAMMXicjZbdTtxGFMed9Cth+5G06lVvRiCkVKJ0F0WqelEpsCSAhNCWAIlSp2jsHdtTbI+ZGbO7WH6H3rav0afJXdXbvkTP2J5jm91WtQQ+/p1z5uN/znjtZTFXejh8d+/+e+9/8OFHDx6uDT7+5NPPHj3+/EKJXPrs3BexkK89qljMU3auuY7Z60wymngxe+VdjY3/1Q2Tiov0TC8y9jahYcoD7lMN6MKde8nlzuWjjeH2sLrIsjFqjA2nuSaXjwdfulPh5wlLtR9TpX4aDTP9tqBScz9m5ZqbK5ZR/
Revised: <latexit sha1_base64="uVoSkjI+4YofA+AhU7/EH/CyZdk=">AAAMMXicjZbdTtxGFMed9Cth+5G06lVvRiCkVKJ0F0WqelEpsCSAhNCWAIlSp2jsHdtTbI+ZGbO7WH6H3rav0afJXdXbvkTP2J5jm91WtQQ+/p1z5uN/znjtZTFXejh8d+/+e+9/8OFHDx6uDT7+5NPPHj3+/EKJXPrs3BexkK89qljMU3auuY7Z60wymngxe+VdjY3/1Q2Tiov0TC8y9jahYcoD7lMN6MKde8nlzuWjjeH2sLrIsjFqjA2nuSaXjwdfulPh5wlLtR9TpX4aDTP9tqBScz9m5ZqbK5ZR/

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Core
Revised: Core

Original: [Table]
Revised: [Table]

Original: y
Revised: y

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +AjxpIsniE+7+BTxeRef/8cCa3k631CVuE80M5rIkMAvsw+fiV/W6eM61/eLMc6P6NShZ4ieoQKIThx6g+gNvlmgus33TDMfkGa91hq3ivYdHU37jlbVvb5jDzujfSnt+04cv1vrwy5ui3rQxe08u11sjxN8SI9ufzavGi93tkffbH/3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit>  Sun et al., 2017).
Revised: +AjxpIsniE+7+BTxeRef/8cCa3k631CVuE80M5rIkMAvsw+fiV/W6eM61/eLMc6P6NShZ4ieoQKIThx6g+gNvlmgus33TDMfkGa91hq3ivYdHU37jlbVvb5jDzujfSnt+04cv1vrwy5ui3rQxe08u11sjxN8SI9ufzavGi93tkffbH/3w87G073mk/qB94W37j3xRt633lPv0Jt4517ghd7P3i/er4PfBr8P/hz8VYfev9fkfOb1rsHf/wBs+XlM</latexit> et al., 2015; Sun et al., 2017).

Original: We numerically validate A SYNC -RED on image recovery from linear and noisy measurements using pre-trained deep denoisers as image priors.
Revised: We numerically validate A SYNC -RED on image recovery from linear and noisy measurements using pre-trained deep denoisers as image priors

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Inverse problems.
Revised: Inverse problems.

Original: Inverse problems are traditionally formulated as a composite optimization problem
Revised: Inverse problems are traditionally formulated as a composite optimization problem

Original: [Equation]
Revised: [Equation]

Original: where g is the data-ﬁdelity term that ensures consistency of x with the measured data y and h is the regularizer that infuses the prior knowledge on x .
Revised: where g is the data-ﬁdelity term that ensures consistency of x with the measured data y and h is the regularizer that infuses the prior knowledge on x .

Original: For example, consider the smooth ` 2 -norm data-ﬁdelity term g ( x )
Revised: For example, consider the smooth ` 2 -norm data-ﬁdelity term g ( x )

Original: =
Revised: =

Original: k y 
Revised: k y

Original: Ax k [Equation] which assumes a linear observation model y
Revised: Ax k 22 , which assumes a linear observation model y

Original: =
Revised: =

Original: Ax + e , and the nonsmooth TV regularizer h ( x )
Revised: Ax + e , and the nonsmooth TV regularizer h ( x )

Original: =
Revised: =

Original: ⌧ k Dx k 1 , where ⌧ > 0 is the regularization parameter and D is the image gradient (Rudin et al., 1992).
Revised: ⌧ k Dx k 1 , where ⌧ > 0 is the regularization parameter and D is the image gradient (Rudin et al., 1992).

Original: Regularization by denoising (RED).
Revised: Regularization by denoising (RED).

Original: RED is a recent methodology for imaging inverse problems that seeks vectors x ⇤ 2 R n satisfying
Revised: RED is a recent methodology for imaging inverse problems that seeks vectors x ⇤ 2 R n satisfying

Original: [Equation]
Revised: [Equation]

Original: where r g denotes the gradient of the data-ﬁdelity term and D  : R n !
Revised: where r g denotes the gradient of the data-ﬁdelity term and D  : R n !

Original: R n is an image denoiser parameterized by  > 0 .
Revised: R n is an image denoiser parameterized by  > 0 .

Original: Under additional technical assumptions, the solutions x ⇤ 2 zer ( G ) can be associated with an explicit objective function of form (1).
Revised: Under additional technical assumptions, the solutions x ⇤ 2 zer ( G ) can be associated with an explicit objective function of form (1).

Original: Speciﬁcally, when D  is locally homogeneous and has a symmetric Jacobian satisfying strong passivity (Romano et al., 2017; Reehorst & Schniter, 2019), H ( x )  corresponds to the gradient of a convex regularizer
Revised: Speciﬁcally, when D  is locally homogeneous and has a symmetric Jacobian satisfying strong passivity (Romano et al., 2017; Reehorst & Schniter, 2019), H ( x ) : = x  D  ( x ) corresponds to the gradient of a convex regularizer

Original: [Equation]
Revised: [Equation]

Original: A simple strategy for computing x
Revised: A simple strategy, known as G M -RED, for computing x

Original: ⇤ 2 zer ( G ) is based on thefollowing ﬁrst-order ﬁxed-point iteration
Revised: ⇤ 2 zer ( G ) is based on the ﬁrst-order ﬁxed-point iteration

Original: [Equation]
Revised: [Equation]

Original: where  > 0 denotes the stepsize.
Revised: where  > 0 denotes the stepsize.

Original: In this paper, we extend this ﬁrst-order RED algorithm to design
Revised: In this paper, we extend this ﬁrst-order RED algorithm to design

Original: A SYNC -RED.
Revised: A SYNC -RED.

Original: Since many denoisers do not satisfy the assumptions necessary for having an explicit objective (Reehorst & Schniter, 2019), our theoretical analysis considers a broader setting where
Revised: Since many denoisers do not satisfy the assumptions necessary for having an explicit objective (Reehorst & Schniter, 2019), our theoretical analysis considers a broader setting where

Original: D  does not necessarily correspond to any explicit regularizer.
Revised: D  does not necessarily correspond to any explicit regularizer.

Original: The beneﬁt of our analysis is that it accommodates powerful deep denoisers (such as DnCNN (Zhang et al., 2017a)) that have been shown to achieve the state-of-the-art performance (Sun et al., 2019a; Wu et al., 2020;
Revised: The beneﬁt of our analysis is that it accommodates powerful deep denoisers (such as DnCNN (Zhang et al., 2017a)) that have been shown to achieve the state-of-the-art performance (Sun et al., 2019a; Wu et al., 2020;

Original: Cohen et al., 2020).
Revised: Cohen et al., 2020).

Original: Plug-and-play priors (PnP) and other related work.
Revised: Plug-and-play priors (PnP) and other related work.

Original: There are other lines of works that combine the iterative methods with advanced denoisers.
Revised: There are other lines of works that combine the iterative methods with advanced denoisers.

Original: One closely-related framework is known as the deep mean-shift priors (Bigdeli et al., 2017).
Revised: One closely-related framework is known as the deep mean-shift priors (Bigdeli et al., 2017).

Original: It develops an implicit regularizer whose gradient is speciﬁed by a denoising autoencoder.
Revised: It develops an implicit regularizer whose gradient is speciﬁed by a denoising autoencoder.

Original: Another well-known framework is PnP, which generalizes proximal methods by replacing the proximal map with an image denoiser (Venkatakrishnan et al., 2013).
Revised: Another well-known framework is PnP, which generalizes proximal methods by replacing the proximal map with an image denoiser (Venkatakrishnan et al., 2013).

Original: Applications and theoretical analysis of PnP are widely studied in (Sreehari et al., 2016; Zhang et al., 2017b;
Revised: Applications and theoretical analysis of PnP are widely studied in (Sreehari et al., 2016; Zhang et al., 2017b;

Original: Sun et al., 2019; Zhang et al., 2019; Ahmad et al., 2020; Wei et al., 2020) and (Chan et al., 2017; Meinhardt et al., 2017; Buzzard et al., 2018; Sun et al., 2019b; Tirer & Giryes, 2019;
Revised: Sun et al., 2019; Zhang et al., 2019; Ahmad et al., 2020; Wei et al., 2020) and (Chan et al., 2017; Meinhardt et al., 2017; Buzzard et al., 2018; Sun et al., 2019b; Tirer & Giryes, 2019;

Original: Teodoro et al., 2019; Ryu et al., 2019; Xu et al., 2020), respectively.
Revised: Teodoro et al., 2019; Ryu et al., 2019; Xu et al., 2020), respectively.

Original: In particular, Buzzard et al.
Revised: In particular, Buzzard et al.

Original: (2018) proposed a parallel extension of PnP called Consensus Equilibrium (CE) , which enables synchronous parallel updates of x .
Revised: (2018) proposed a parallel extension of PnP called Consensus Equilibrium (CE) , which enables synchronous parallel updates of x .

Original: Note that while we developed A SYNC -RED as a variant of RED, our framework and analysis can be also potentially applied to PnP/CE.
Revised: Note that while we developed A SYNC -RED as a variant of RED, our framework and analysis can be also potentially applied to PnP/CE.

Original: The plug-in strategy can be also applied to another family of algorithms known as approximate message passing (AMP)
Revised: The plug-in strategy can be also applied to another family of algorithms known as approximate message passing (AMP)

Original: (Metzler et al., 2016a;b; Fletcher et al., 2018).
Revised: (Metzler et al., 2016a;b; Fletcher et al., 2018).

Original: The AMP-based algorithms are known to be nearly-optimal for random measurement matrices, but are generally unstable for general A (Rangan et al., 2014; 2015).
Revised: The AMP-based algorithms are known to be nearly-optimal for random measurement matrices, but are generally unstable for general A (Rangan et al., 2014; 2015).

Original: Asynchronous parallel optimization.
Revised: Asynchronous parallel optimization.

Original: There are two main lines of work in asynchronous parallel optimization, the one involving the asynchrony in coordinate updates (Liu et al., 2015; Peng et al., 2016;  Sun et al., 2017; Hannah & Yin, 2018;
Revised: There are two main lines of work in asynchronous parallel optimization, the one involving the asynchrony in coordinate updates (Iutzeler et al., 2013; Liu et al., 2015; Peng et al., 2016; Bianchi et al., 2015; Sun et al., 2017; Hannah & Yin, 2018;

Original: Hannah et al., 2019), and the other focusing on the study of various asynchronous stochastic gradient methods (Recht et al., 2011; Lian et al., 2015;
Revised: Hannah et al., 2019), and the other focusing on the study of various asynchronous stochastic gradient methods (Recht et al., 2011; Lian et al., 2015;

Original: Liu et al., 2018;
Revised: Liu et al., 2018;

Original: Zhou et al., 2018; Lian et al., 2018).
Revised: Zhou et al., 2018; Lian et al., 2018).

Original: Our work contributes to the area by developing a novel deep-regularized asynchronous parallel method with provable convergence guarantees.
Revised: Our work contributes to the area by developing a novel deep-regularized asynchronous parallel method with provable convergence guarantees.

Original:  A SYNCHRONOUS RED
Revised:  A SYNCHRONOUS RED

Original: A SYNC -RED addresses the computational bottleneck by simultaneously considering the asynchronous partial updates of image x and the  randomized usage of measurements y .
Revised: A SYNC -RED allows efﬁcient processing of data by simultaneously considering the asynchronous partial updates of solution x and the use of randomized subset of measurements y .

Original: In this section, we introduce the algorithmic details of our method.
Revised: In this section, we introduce the algorithmic details of our method.

Original: We start with the basic batch formulation of
Revised: We start with the basic batch formulation of

Original: A SYNC -RED
Revised: A SYNC -RED

Original: (A SYNC -RED-BG) followed by its minibatch variant (A SYNC -RED-SG).
Revised: (A SYNC -RED-BG) followed by its minibatch variant (A SYNC -RED-SG).

Original: 3. A SYNC -RED USING B ATCH G RADIENT
Revised: 3. A SYNC -RED USING B ATCH G RADIENT

Original: When the gradient uses all the measurements y 2 R m , A SYNC -RED-BG is the asynchronous extension of the recent block-coordinate RED (BC-RED) algorithm
Revised: When the gradient uses all the measurements y 2 R m , A SYNC -RED-BG is the asynchronous extension of the recent block-coordinate RED (BC-RED) algorithm

Original: (Sun et al., 2019a).
Revised: (Sun et al., 2019a).

Original: Consider the decomposition of the variable space R n into b  1 blocks
Revised: Consider the decomposition of the variable space R n into b  1 blocks

Original: [Equation]
Revised: [Equation]

Original: with n = n 1 + n 2 +· · · + n b
Revised: with n = n 1 + n 2 +

Original: [Equation]
Revised: [Equation]

Original: For each i 2 { 1 , . . .
Revised: For each i 2 { 1 , . . .

Original: , b } , we introduce the operator U i : R n i !
Revised: , b } , we introduce the operator U i : R n i !

Original: R n that injects a vector in R n i into
Revised: R n that injects a vector in R n i into

Original: R n and its transpose U T i that extracts the i th block from a vector in R n .
Revised: R n and its transpose U T i that extracts the i th block from a vector in R n .

Original: This directly implies that
Revised: This directly implies that

Revised: I

Original: [Equation]
Revised: [Equation]

Original: In analogy to the RED operator G in (2), we deﬁne the block-coordinate operator G i as
Revised: and k x k [Equation] = k

Original: [Equation]
Revised: [Equation]

Original: Due to the asynchrony in  the block updates, the iterate might be updated several times by different cores during a single update cycle of a core, which means that the evaluation of x k +1 relies on a stale iterate
Revised: k x b kwith x i = U T i x . In analogy to the RED operator G in (2), we deﬁne the block-coordinate operator G i as

Original: [Equation]
Revised: [Equation]

Original: ( e x k ) , with 
Revised: i [Equation] , with x 2 R n and G i : R n ! R n .

Revised: Due to the asynchrony in the block updates, the iterate might be updated several times by different cores during a single update cycle of a core, which means that the evaluation of x k +1 relies on a stale iterate

Original: [Equation]
Revised: [Equation]

Original:  Here, we assume that the stale iterate e x k exits as a state of x in the shared memory, and the delay between them is bounded by a ﬁnite number  2 Z + .
Revised: with [Equation] Here, we assume that the stale iterate e x k exits as a state of x in the shared memory, and the delay between them is bounded by a ﬁnite number  2 Z + .

Original: These two assumptions are often referred to as the consistent read (Recht et al., 2011) and the bounded delay (Liu & Wright, 2015) in the traditional asynchronous block coordinate optimization.
Revised: These two assumptions are often referred to as the consistent read (Recht et al., 2011) and the bounded delay (Liu & Wright, 2015) in the traditional asynchronous block coordinate optimization.

Original: Although we implement the consistent read in A SYNC -RED, the algorithm never imposes a global lock on x k .
Revised: Although we implement the consistent read in A SYNC -RED, the algorithm never imposes a global lock on x k .

Original: We refer to Supplement A for the related discussion.
Revised: We refer to Supplement A for the related discussion.

Original: We now introduce the ﬁrst variant, A SYNC -RED-BG.
Revised: The ﬁrst variant, A SYNC -RED-BG, is summarized in Algorithm 1, where read ( · ) reads a block from the shared memory to the local memory.

Original: When the algorithm is run on a single core system without  parallelization (that is to say e x k = x k ), it reduces to the normal BC-RED algorithm.
Revised: When the algorithm is run on a single core system without 6: x 7: x x G e memory parallelization (that is to say e x k = x k ), it reduces to the normal BC-RED algorithm.

Original: Hence, our analysis is also applicable to BC-RED.
Revised: Hence, our analysis is also applicable to BC-RED.

Original: We speciﬁcally consider the random block selection strategy in A SYNC -RED-BG, namely that every block index i k is selected as an i.i.d random variable uniformly distributed over { 1 , . . .
Revised: We speciﬁcally consider the random block selection strategy in A SYNC -RED-BG, namely that every block index i k is selected as an i.i.d random variable uniformly distributed over { 1 , . . .

Original: , b } .
Revised: , b } .

Original: Such a strategy is commonly adopted for simplifying the convergence analysis.
Revised: Such a strategy is commonly adopted for simplifying the convergence analysis.

Original: Nevertheless, our method and analysis can be generalized to the scenario where i k follows some arbitrary probability
Revised: Nevertheless, our method and analysis can be generalized to the scenario where i k follows some arbitrary probability

Original: P ( i k
Revised: P ( i k

Original: = i ) =
Revised: = i ) =

Original: p i speciﬁed by the user.
Revised: p i speciﬁed by the user.

Original: Compared with serial RED algorithms, A SYNC -RED-BG enjoys considerable scalability by dividing the computation of the full operator G into b parallel evaluation of G i distributed across all cores.
Revised: Compared with serial RED algorithms, A SYNC -RED-BG enjoys considerable scalability by dividing the computation of the full operator G into b parallel evaluation of G i distributed across all cores.

Original: Thus, without any modiﬁcation to the algorithmic design, one can easily improve the performance of the algorithm by simply integrating more cores into the system.
Revised: Thus, without any modiﬁcation to the algorithmic design, one can easily improve the performance of the algorithm by simply integrating more cores into the system.

Original: In Section 5, we experimentally demonstrate the signiﬁcant speed-up and scale-up in solving the context of image recovery.
Revised: In Section 5, we experimentally demonstrate the signiﬁcant speed-up and scale-up in solving the context of image recovery.

Original: 3. A SYNC -RED USING S TOCHASTIC G RADIENT
Revised: 3. A SYNC -RED USING S TOCHASTIC G RADIENT

Original: The scale of measurements is another important factor inﬂuencing the computational complexity in the large-scale inference tasks.
Revised: The scale of measurements is another important factor inﬂuencing the computational complexity in the large-scale inference tasks.

Original: A SYNC -RED-SG improves the applicability of A SYNC -RED to these cases by further considering the decomposition of the measurement space R m into `  1 blocks 
Revised: A SYNC -RED-SG improves the applicability of A SYNC -RED to these cases by further considering the decomposition of the measurement space R m into `  1 blocks y

Original: [Equation]
Revised: [Equation]

Original: with m = m 1 + m [Equation] m ` .
Revised: with m = m 1 + m 2 + · · · + m ` .

Original: Hence, A SYNC -RED-SG considers the following data-ﬁdelity g and its gradient r g
Revised: Hence, A SYNC -RED-SG considers the following data-ﬁdelity g and its gradient r g

Original: [Equation]
Revised: [Equation]

Original: where each g j is evaluated on the subset y j 2 R m j of the full y .
Revised: where each g j is evaluated on the subset y j 2 R m j of the full y .

Original: From (8), we know that the computation of r g ( x ) is proportional to the total number ` .
Revised: From (8), we know that the computation of r g ( x ) is proportional to the total number ` .

Original: To reduce the per-iteration cost, we follow the idea of stochastic optimization to approximate the batch gradient by using the stochastic gradient that relies on a minibatch of w ⌧ ` measurements
Revised: To reduce the per-iteration cost, we follow the idea of stochastic optimization to approximate the batch gradient by using the stochastic gradient that relies on a minibatch of w ⌧ ` measurements

Original: [Equation]
Revised: [Equation]

Original: where j s is picked from the set { 1 , .
Revised: where j s is picked from the set { 1 , .

Original: , ` } as i.i.d uniform random variable.
Revised: , ` } as i.i.d uniform random variable.

Original: Based on the minibatch gradient, we deﬁne the block stochastic operator
Revised: Based on the minibatch gradient, we deﬁne the block stochastic operator

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: nas
Revised: n

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Note that the computation of b
Revised: Note that the computation of b

Original: G i is now dependent on the minibatch size w that is adjustable to cope with the computation resources at hand.
Revised: G i is now dependent on the minibatch size w that is adjustable to cope with the computation resources at hand.

Original: A SYNC -RED-SG is summarized in Algorithm 2.
Revised: A SYNC -RED-SG is summarized in Algorithm 2.

Revised: The operation minibatchG ( · ) computes the estimate of G based on w randomly selected measurements.

Original: We clarify the difference between A SYNC -RED-SG and A SYNC -RED-BG via a speciﬁc example.
Revised: We clarify the difference between A SYNC -RED-SG and A SYNC -RED-BG via a speciﬁc example.

Original: Consider the least-squares g with a block-friendly operator A and a block-efﬁcient denoiser D  .
Revised: Consider the least-squares g with a block-friendly operator A and a block-efﬁcient denoiser D  .

Original: We can write the update of A SYNC -RED-BG regarding a single iteration as
Revised: We can write the update of A SYNC -RED-BG regarding a single iteration as

Original: [Equation]
Revised: [Equation]

Original: where e x is the delayed iterate for  x , and
Revised: i [Equation] 1: input: x 0 2 R n ,  > 0 , ⌧ > 0 . 3: for do 4: e x k read

Revised: ) 5: b

Revised: G ( e x k ) minibatchG ( e x k , w ) with random j w 2 { 1 , . . .

Revised: , ` } .

Revised: Minibatch Gradient 6: b

Revised: G i k

Revised: ( e x k )

Original: A
Revised: U

Revised: i k

Revised: U T

Original: i 2 R m ⇥ n i is a submatrix of A consisting of columns corresponding to the i th blocks.
Revised: i k b G ( e x k ) with random i k 2 { 1 , . . . , b }

Revised: 7: x k read

Revised: ) 8: x k + x k   b

Revised: G i k

Revised: ( e x k ) 9: update x the shared

Revised: + 10: end for where e x is the delayed iterate for x , and A i 2 R m ⇥ n i is a submatrix of A consisting of columns corresponding to the i th blocks.

Original: Although the per-iteration complexity is reduced by roughly b = n/n i times by working with A i instead of A , A SYNC -RED-BG still needs to work with all the measurements y i related to the i th block at every iteration.
Revised: Although the per-iteration complexity is reduced by roughly b = n/n i times by working with A i instead of A , A SYNC -RED-BG still needs to work with all the measurements y i related to the i th block at every iteration.

Original: Consider the corresponding update of
Revised: Consider the corresponding update of

Original: A SYNC -RED-SG with one measurement used at a time
Revised: A SYNC -RED-SG with one measurement used at a time

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where y ji denotes the j th measurement of x i , and A ji 2 R m j ⇥ n i is the submatrix crossed by the rows and columns corresponding to the j th measurement and the i th blocks.
Revised: where y ji denotes the j th measurement of x i , and A ji 2 R m j ⇥ n i is the submatrix crossed by the rows and columns corresponding to the j th measurement and the i th blocks.

Original: This indicates that the reduction of the per-iteration complexity from A SYNC -RED-BG to A SYNC -RED-SG can be up to ` = m/m j times.
Revised: This indicates that the reduction of the per-iteration complexity from A SYNC -RED-BG to A SYNC -RED-SG can be up to ` = m/m j times.

Original: In the practice, it is common to use w > 1 measurements at a time to optimize the total runtime.
Revised: In the practice, it is common to use w > 1 measurements at a time to optimize the total runtime.

Original: Note that if U = U
Revised: Note that if U = U

Original: T
Revised: T

Original: =
Revised: =

Original: I , A SYNC -RED-SG becomes the asynchronous stochastic RED algorithm.
Revised: I , A SYNC -RED-SG becomes the asynchronous stochastic RED algorithm.

Original: In the next section, we will present a complete analysis of A SYNC -RED and theoretically discuss its connection to the related algorithms.
Revised: In the next section, we will present a complete analysis of A SYNC -RED and theoretically discuss its connection to the related algorithms.

Original:  C ONVERGENCE A NALYSIS OF A SYNC -RED
Revised:  C ONVERGENCE A NALYSIS OF A SYNC -RED

Original: The proposed analysis is based on the following explicit assumptions.
Revised: The proposed analysis is based on the following explicit assumptions.

Original: Note that these assumptions serve as sufﬁcient conditions for the convergence.
Revised: Note that these assumptions serve as sufﬁcient conditions for the convergence.

Original: Assumption 1.
Revised: Assumption 1.

Original: We assume bounded maximal delay  < 1 .
Revised: We assume bounded maximal delay  < 1 .

Original: Hence, during any update cycle of an agent, the estimate x in the shared memory is updated at most  2 Z + times by other cores.
Revised: Hence, during any update cycle of an agent, the estimate x in the shared memory is updated at most  2 Z + times by other cores.

Original: The value of  is often dependent on the number of cores involved in the computation (Wright, 2015).
Revised: The value of  is often dependent on the number of cores involved in the computation (Wright, 2015).

Original: If every core takes a similar amount of time to compute its update,  is expected to be a multiple of the number of cores.
Revised: If every core takes a similar amount of time to compute its update,  is expected to be a multiple of the number of cores.

Original: Related work has investigated the convergence with unbounded maximal delays in the context of traditional optimization (Hannah & Yin, 2018; Peng et al., 2019;
Revised: Related work has investigated the convergence with unbounded maximal delays in the context of traditional optimization (Hannah & Yin, 2018; Peng et al., 2019;

Original: Zhou et al., 2018).
Revised: Zhou et al., 2018).

Original: Assumption 2.
Revised: Assumption 2.

Original: The operator G is such that zer ( G ) 

Revised: The operator G is such that zer ( G )

Original: = ? , and the distance of the initial x 0 2 R n to any element in zer ( G ) is bounded, that is k x 

Revised: = ? , and the distance of the initial x 0 2 R n to any element in zer ( G ) is bounded, that is k x

Original:  x ⇤ k  R 0 for all x ⇤ 2 zer ( G ) with R 0 < 1 .
Revised:  x ⇤ k  R 0 for all x ⇤ 2 zer ( G ) with R 0 < 1 .

Original: This assumption ensures the existence of a solution for the RED problem and is related to the existence of minimizers in traditional coordinate minimization (Nesterov, 2012; Beck & Tetruashvili, 2013)
Revised: This assumption ensures the existence of a solution for the RED problem and is related to the existence of minimizers in traditional coordinate minimization (Nesterov, 2012; Beck & Tetruashvili, 2013)

Original: Assumption 3.
Revised: Assumption 3.

Original: (a) Every component function g
Revised: (a) Every component function g

Original: i is convex differentiable and has a Lipschitz continuous gradient of constant L i > 0 .
Revised: i is convex differentiable and has a Lipschitz continuous gradient of constant L i > 0 .

Original: (b) At every update, the stochastic gradient is unbiased estimator of r g that has a bounded variance:
Revised: (b) At every update, the stochastic gradient is unbiased estimator of r g that has a bounded variance:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst part of the assumption implies that g is also convex and has Lipschitz continuous gradient with constant L = max { L 1 , . . .
Revised: The ﬁrst part of the assumption implies that g is also convex and has Lipschitz continuous gradient with constant L = max { L 1 , . . .

Original: , L ` } .
Revised: , L ` } .

Original: The second part is a standard assumption on the unbiasedness 
Revised: The second part is a standard assumption on the unbiasedness and variance of the stochastic gradient (Lian et al., 2015; Ghadimi & Lan, 2016). Our ﬁnal assumption is related to the deep denoiser used in A SYNC -RED. Assumption 4. The denoiser D  is a nonexpansive operator k D  ( x )  D  ( y ) k  k x  y k .

Original: [Table]
Revised: [Table]

Original: S N R ( d
Revised: S N R ( d

Original: B )
Revised: B )

Original: [Table]
Revised: [Table]

Original: /KhSMFkGxxF0Yqj83+Jo3OMi3i8FAQM/FAeMLiu4lZNWs1H6oAVs1UT1QEqS5cCgOFKeOIv+YGhH4o/R4d5wC2IoFm6CCy+YxIx2Bb7vBkFbIunykc8gUa0/CAZjkZU6TyfaDbXeflcvJsAKurSeXBKNYVXcJVxYpTIT66a4TRPFqTSok7pvOLKcrkxzAD/KO7XtchF5FnkIWIWMUShRSEiZZFCNLeo6YXAogDRwqIFojuL7hBJiySiG4tuECUWNV2aWZQhOrHoBNGtRbeIZhbNEImYBShj9YQ+OHi68ZVPRatPhKHw17dsz8buYdiBRQeIji06XlVYMoNfBQJ9wiSfo+ZVK5Wy2yayQqNr0XXdtVx33dZbmi6kGqcCG5JmfMqMZUtdZyqa1P16Di2KvXqOmztr47NGhjZupHjZxi8RH7XxEeJRG48QH7bxYSNzGx8jHrfxGPFpG58ivmjji/9YYCVP6xuqFHdLMa2I8An8MrvwsfisSh9Wua6bD3F+RKcWvUD0AhVANLLoDaI3+GaB6tbfM/V8QOr1GmvYKNp1tDTtOhpV97uOfeyM5qV04Fpx3Hatj9q4KephGzfz7LWxOU7wIT24/9m8bLza3Rl8u/P9j7sbz/frT+pHzlfOurPlDJzvnOfOkTN2LhzPiZyfnV+cX3u/9f7o/dn7qwp9+KDO+cLpXL2//wGxJXq7</latexit>
Revised: /KhSMFkGxxF0Yqj83+Jo3OMi3i8FAQM/FAeMLiu4lZNWs1H6oAVs1UT1QEqS5cCgOFKeOIv+YGhH4o/R4d5wC2IoFm6CCy+YxIx2Bb7vBkFbIunykc8gUa0/CAZjkZU6TyfaDbXeflcvJsAKurSeXBKNYVXcJVxYpTIT66a4TRPFqTSok7pvOLKcrkxzAD/KO7XtchF5FnkIWIWMUShRSEiZZFCNLeo6YXAogDRwqIFojuL7hBJiySiG4tuECUWNV2aWZQhOrHoBNGtRbeIZhbNEImYBShj9YQ+OHi68ZVPRatPhKHw17dsz8buYdiBRQeIji06XlVYMoNfBQJ9wiSfo+ZVK5Wy2yayQqNr0XXdtVx33dZbmi6kGqcCG5JmfMqMZUtdZyqa1P16Di2KvXqOmztr47NGhjZupHjZxi8RH7XxEeJRG48QH7bxYSNzGx8jHrfxGPFpG58ivmjji/9YYCVP6xuqFHdLMa2I8An8MrvwsfisSh9Wua6bD3F+RKcWvUD0AhVANLLoDaI3+GaB6tbfM/V8QOr1GmvYKNp1tDTtOhpV97uOfeyM5qV04Fpx3Hatj9q4KephGzfz7LWxOU7wIT24/9m8bLza3Rl8u/P9j7sbz/frT+pHzlfOurPlDJzvnOfOkTN2LhzPiZyfnV+cX3u/9f7o/dn7qwp9+KDO+cLpXL2//wGxJXq7</latexit>

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 29.00 dB
Revised: 29.00 dB

Original: [Figure]
Revised: [Figure]

Original: latexit sha1_base64="9ZZ/bPBwBFHpWjncVOEQW4JPq7g=">AAAMTHicjZZbb9s2FMfVbuvauNvSDXvaw4ikAVIgyezs0uahQBM7TQIEgZdLW3TuAkqmJCISqZBUbEfQ4z7NXrevsfd9j70NA3YoiZSUuMME2CJ/5xxe
Revised: latexit sha1_base64="9ZZ/bPBwBFHpWjncVOEQW4JPq7g=">AAAMTHicjZZbb9s2FMfVbuvauNvSDXvaw4ikAVIgyezs0uahQBM7TQIEgZdLW3TuAkqmJCISqZBUbEfQ4z7NXrevsfd9j70NA3YoiZSUuMME2CJ/5xxe

Original: [Table]
Revised: [Table]

Original: +6/3R+7P3V2V6/17t86XTeXp//wOfJH5I</latexit> iteration ( k ) 500 1010 500 1010k G ( x k ) k/ k G ( x 0 ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv/2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU/Xix86yN+iW6XNvob/fLiywag9rYcOprdPlk5Vt3LPwsZon2I6rUL4N+qj/kVGruR6xYdjPFUupf0ZDlnhBXmnqqINXVI6kUAVNm0TT6/jqjEddzAiERU51MGqtAJLqV6UXg8gSVYxJTPSFqHnsiWsgy2CZB1g2VidATnoSdwIT7LJDUb4b3RQxObYclgZBk8GxnizDtb3dyY+5LYRQvbK4lIpQ0ncy7EsTFY7h6hGvYrE9g8HEnwOeaFWVEKEWWEnguK9jdWTky92eFCQx4mMk7ghlNtICVVxGw+nXTQwm7vl7vahSFQnI9gUUXEImP6v6o4q7AhpXL1RMmJLuTx5Ishg3F5Sq0IBN6w0jEtGaSGB
Revised: +6/3R+7P3V2V6/17t86XTeXp//wOfJH5I</latexit> iteration ( k ) 500 1010 500 1010k G ( x k ) k/ k G ( x 0 ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv/2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU/Xix86yN+iW6XNvob/fLiywag9rYcOprdPlk5Vt3LPwsZon2I6rUL4N+qj/kVGruR6xYdjPFUupf0ZDlnhBXmnqqINXVI6kUAVNm0TT6/jqjEddzAiERU51MGqtAJLqV6UXg8gSVYxJTPSFqHnsiWsgy2CZB1g2VidATnoSdwIT7LJDUb4b3RQxObYclgZBk8GxnizDtb3dyY+5LYRQvbK4lIpQ0ncy7EsTFY7h6hGvYrE9g8HEnwOeaFWVEKEWWEnguK9jdWTky92eFCQx4mMk7ghlNtICVVxGw+nXTQwm7vl7vahSFQnI9gUUXEImP6v6o4q7AhpXL1RMmJLuTx5Ishg3F5Sq0IBN6w0jEtGaSGB

Original: [Table]
Revised: [Table]

Original: +JYTMLBDeqBBVem3aNJyPlQ5LJitnKhykGlyywGYXQll/i07MGuH5M+sQXfsFnhQL50HBl8TYTG0DfZpPQq0DZ5I3+IxFKLhQzY4OsJSZVnxYsiKfv7zGFBepc6DU6owPILLiEOtRHZ4Xg+nKJujUosqpPWIK9LlxjAD/GG7X9cg1yLPIM8iYhCxKDQotEgaJC2aGVTXQmBQYNHcoLlF1wZdWyQMEhZdGnRpETOortLUoNSiQ4MOLboy6MqiqUFTi3hMAitj2bM2OHiqthW9vFEnXFP49QzbNr7b1m1o0NCiA4MOFiUWTeGtgKBOiKAzq3lZSoXspoiM0NY0b5uuG6brdundmi7Eyk4FbQia0gnRLZPqKlJiVtXrCZSordUTu7lXTfyqlqGJayleNvFLi/ebeN/ioyY+snivifdqmZv4wOJRE48sPm7iY4tPm/j0PxZYytP4hirEfSyJkoj7CN7MLnw+PinDB2Ws62YDO79FxwbtWrRrFbDoyKA3Fr2xTxbIbvU9U80HpFqvbg1qRduGhqZtQ63qTtuwYyujfigNXSOO28z1fhPXSd1r4nqe7SbWxwk+pM3XMnp342yj29/s9vs/bKy+2Km+qe85XzkrzmOn72w5L5x9Z+ScOp7zi/Or85vze+ePzl+dvzv/lK5371Qxnzut64HzL4zlgLI=</latexit> BC-RED ( n c =1) 8000 10101010 8000 time (second) k G ( x k )
Revised: +JYTMLBDeqBBVem3aNJyPlQ5LJitnKhykGlyywGYXQll/i07MGuH5M+sQXfsFnhQL50HBl8TYTG0DfZpPQq0DZ5I3+IxFKLhQzY4OsJSZVnxYsiKfv7zGFBepc6DU6owPILLiEOtRHZ4Xg+nKJujUosqpPWIK9LlxjAD/GG7X9cg1yLPIM8iYhCxKDQotEgaJC2aGVTXQmBQYNHcoLlF1wZdWyQMEhZdGnRpETOortLUoNSiQ4MOLboy6MqiqUFTi3hMAitj2bM2OHiqthW9vFEnXFP49QzbNr7b1m1o0NCiA4MOFiUWTeGtgKBOiKAzq3lZSoXspoiM0NY0b5uuG6brdundmi7Eyk4FbQia0gnRLZPqKlJiVtXrCZSordUTu7lXTfyqlqGJayleNvFLi/ebeN/ioyY+snivifdqmZv4wOJRE48sPm7iY4tPm/j0PxZYytP4hirEfSyJkoj7CN7MLnw+PinDB2Ws62YDO79FxwbtWrRrFbDoyKA3Fr2xTxbIbvU9U80HpFqvbg1qRduGhqZtQ63qTtuwYyujfigNXSOO28z1fhPXSd1r4nqe7SbWxwk+pM3XMnp342yj29/s9vs/bKy+2Km+qe85XzkrzmOn72w5L5x9Z+ScOp7zi/Or85vze+ePzl+dvzv/lK5371Qxnzut64HzL4zlgLI=</latexit> BC-RED ( n c =1) 8000 10101010 8000 time (second) k G ( x k )

Original: k/ k G ( x 0 ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv/2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU
Revised: k/ k G ( x 0 ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv/2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU

Original: [Table]
Revised: [Table]

Original: /MPRD8WfoMA+4BRE2SxehxbdMIgbb4oA3o4Bt8VgFiF1oRMv3k+HJCVU6z13NZjovn4sLF1BRl86HU6opvIKrjGOjRH582QyneTInlRZ1SucVV5bLi2EG+Edxv55FHiLfIh8Rs4ghmlg0QaQsUohmFjW9EFoUIppbNEd0a9EtImmRRHRt0TWixKKmSzOLMkTHFh0jurHoBtHUoikiEbMQZaye0AcHTze+8qlo9YkwFP76lu3a2F0M27doH9GRRUf3FZZM4VeBQJ8wyWeoedVKpey2iazQ6Jp3Xbct12239Ramm1CNU4ENSVM+Zsaypa4zFU3qfj2DFsVePcPNvW7j140MbdxI8aqNXyE+bONDxCdtfIL4oI0PGpnb+AjxqI1HiE/b+BTxeRuf/8cCK3la31CluJuKaUVEQOCX2YNvyadV+rDK9bx8iPMjOrXoJaKXqACiE4veI3qPbxaobv09U88HpF6vsYaNol1HS9Ouo1F1r+vYw85oXkr7nhXHa9f6sI2boh60cTPPbhub4wQf0oO7n82Lxpud7cHz7R9/fr7xYq/+pH7kfOesO5vOwPnBeeEcOiPn3PGdX53fnN+dP1b+XPl7dWn1YRX6YKnO+cbpXKur/wBgkYPv</latexit> 

Revised: /MPRD8WfoMA+4BRE2SxehxbdMIgbb4oA3o4Bt8VgFiF1oRMv3k+HJCVU6z13NZjovn4sLF1BRl86HU6opvIKrjGOjRH582QyneTInlRZ1SucVV5bLi2EG+Edxv55FHiLfIh8Rs4ghmlg0QaQsUohmFjW9EFoUIppbNEd0a9EtImmRRHRt0TWixKKmSzOLMkTHFh0jurHoBtHUoikiEbMQZaye0AcHTze+8qlo9YkwFP76lu3a2F0M27doH9GRRUf3FZZM4VeBQJ8wyWeoedVKpey2iazQ6Jp3Xbct12239Ramm1CNU4ENSVM+Zsaypa4zFU3qfj2DFsVePcPNvW7j140MbdxI8aqNXyE+bONDxCdtfIL4oI0PGpnb+AjxqI1HiE/b+BTxeRuf/8cCK3la31CluJuKaUVEQOCX2YNvyadV+rDK9bx8iPMjOrXoJaKXqACiE4veI3qPbxaobv09U88HpF6vsYaNol1HS9Ouo1F1r+vYw85oXkr7nhXHa9f6sI2boh60cTPPbhub4wQf0oO7n82Lxpud7cHz7R9/fr7xYq/+pH7kfOesO5vOwPnBeeEcOiPn3PGdX53fnN+dP1b+XPl7dWn1YRX6YKnO+cbpXKur/wBgkYPv</latexit>

Original: S yn c - R E D ( 8 - c o r e ) <latexit sha1_base64="J3bg65h54GDAwZq87E+RuqOxJ3A=">AAAMRnicjZbdbts2FMfV7quNuy3dsIthN0SCACmQerbTZdnFgCZ2mgQIAi8fbdG6CyiZkohIokJSsRVB2NPsdnuNvcJeYnfDbncoiZSUuMME2CJ/5xx+/M+hJDsOqJC93p/37n/w4Ucff/Lg4VLn0aeffb78+IuXgiXcIecOCxh/bWNBAhqRc0llQF7HnODQDsgr+3Ko7K+uCReURWcyjcm7EHsRdamDJaCL5a8nksylcLLTNHKenuyN8vXtpw7j5MnF8mqvu7W9vflsgHrdwdbWZl81eptbg+/6qN/tFdeqVV3ji8edryZT5iQhiaQTYCHe9nuxfJdhLqkTkHxpkggSY+cSeySzGbuU2BY5Kq81FHPmEqEWioOnVwkOqEwRuAREtCJxKFwWyUakHYDJZphPUYilj0Qa2iy4E6WwDoKoa8wjJn0aeS3HiDrE5diph3dYCEaph0Uu46j/7WADEel0W7EhdThTKuc6VhPmcRz7aVuCMH8I1xqiEjbrIBh82nJwqCR54eFxlsQI+kXW2jsrRqbOPFeOLvUSfkswpYlksPLSA1a/ouomIldXK22NAo9xKn1YdA6episWe+W3BVasWK70CVTQrTgSJSFsKCxWIRny8TVBAZGScKRshC/YXJgEknI2q7MBkR6JlDNZVB2AkgBzUON/ufu+Oji3dsLTqfCDqFjolMECp0h5CUQjhMtRWgHz4hjmS0tra2tozBlzYRKtgUwDksUBplGuilw10I9VyjfQlLg0omrfADkLcbQ0icisCs7gTMRMFPY8Gzc6La96kDwb1e2WT3XPs7Oq0bIGJAxxnh0Vt5YFHgUsAEHTPBuaZssDDnoSxuXsO3W7PQpsG+YeFrdSJxarpDAOUsH61S/KJhsTXZmTjbw0YO6FFEyqillc3oULSfImG2DI84Yfnr/HD8+NX0DDO07AwA7pgQaVpd+iScv5UOWwYLZyospBJPEdB2BmJTRy79iBGTskf24MqmO2wLx66czT+IZwg6GtsUvrUaCt8VS4Bk+gEDUfRcPjYyxklhVvhqzo5z9PAOVV6hw4pRLDI7iMOFJKZEcX9XCSRikqtahCWo+4Il12CDPAHzb7tTWyDXI0cgwiGhGDfI18g4RGwqC5RnUteBp5BqUapQbdaHRjENeIG3Sl0ZVBkUZ1lSYaJQYdaXRk0LVG1wbNNJoZxELiGRnLnrHBwZO1rejljTphisKvp9mO9t0xbiONRgYdanS4KLFoBm8FBHVCOJ0bzctSKmTXRaSFNqa0bbppmG7apXdnOh9LMxW0IWhGp0S1dKqrSIGjql7PoERNrZ6ZzZ028WktQxPXUrxo4hcGHzTxgcHHTXxs8H4T79cyN/GhweMmHht80sQnBp838fl/LLCUp/ENVYi7LogUiLkI3sw2fD8+KcOHZaxtZ0Mzv0EnGu0ZtGcUMOhYozcGvTFPFshu9T1TzQekWq9qDWtF24aGpm1Drepu27BrKqN+KI1sLY7dzPVBE9dJ3W/iep6dJlbHCT6k9dcyen/j5aDbf9b94afB6vPd6pP6gfWNtWKtW33re+u5dWCNrXPLsX6xfrV+s37v/NH5q/N355/S9f69KuZLq3U9sv4FgkaBxA==</latexit>
Revised: S yn c - R E D ( 8 - c o r e ) <latexit sha1_base64="J3bg65h54GDAwZq87E+RuqOxJ3A=">AAAMRnicjZbdbts2FMfV7quNuy3dsIthN0SCACmQerbTZdnFgCZ2mgQIAi8fbdG6CyiZkohIokJSsRVB2NPsdnuNvcJeYnfDbncoiZSUuMME2CJ/5xx+/M+hJDsOqJC93p/37n/w4Ucff/Lg4VLn0aeffb78+IuXgiXcIecOCxh/bWNBAhqRc0llQF7HnODQDsgr+3Ko7K+uCReURWcyjcm7EHsRdamDJaCL5a8nksylcLLTNHKenuyN8vXtpw7j5MnF8mqvu7W9vflsgHrdwdbWZl81eptbg+/6qN/tFdeqVV3ji8edryZT5iQhiaQTYCHe9nuxfJdhLqkTkHxpkggSY+cSeySzGbuU2BY5Kq81FHPmEqEWioOnVwkOqEwRuAREtCJxKFwWyUakHYDJZphPUYilj0Qa2iy4E6WwDoKoa8wjJn0aeS3HiDrE5diph3dYCEaph0Uu46j/7WADEel0W7EhdThTKuc6VhPmcRz7aVuCMH8I1xqiEjbrIBh82nJwqCR54eFxlsQI+kXW2jsrRqbOPFeOLvUSfkswpYlksPLSA1a/ouomIldXK22NAo9xKn1YdA6episWe+W3BVasWK70CVTQrTgSJSFsKCxWIRny8TVBAZGScKRshC/YXJgEknI2q7MBkR6JlDNZVB2AkgBzUON/ufu+Oji3dsLTqfCDqFjolMECp0h5CUQjhMtRWgHz4hjmS0tra2tozBlzYRKtgUwDksUBplGuilw10I9VyjfQlLg0omrfADkLcbQ0icisCs7gTMRMFPY8Gzc6La96kDwb1e2WT3XPs7Oq0bIGJAxxnh0Vt5YFHgUsAEHTPBuaZssDDnoSxuXsO3W7PQpsG+YeFrdSJxarpDAOUsH61S/KJhsTXZmTjbw0YO6FFEyqillc3oULSfImG2DI84Yfnr/HD8+NX0DDO07AwA7pgQaVpd+iScv5UOWwYLZyospBJPEdB2BmJTRy79iBGTskf24MqmO2wLx66czT+IZwg6GtsUvrUaCt8VS4Bk+gEDUfRcPjYyxklhVvhqzo5z9PAOVV6hw4pRLDI7iMOFJKZEcX9XCSRikqtahCWo+4Il12CDPAHzb7tTWyDXI0cgwiGhGDfI18g4RGwqC5RnUteBp5BqUapQbdaHRjENeIG3Sl0ZVBkUZ1lSYaJQYdaXRk0LVG1wbNNJoZxELiGRnLnrHBwZO1rejljTphisKvp9mO9t0xbiONRgYdanS4KLFoBm8FBHVCOJ0bzctSKmTXRaSFNqa0bbppmG7apXdnOh9LMxW0IWhGp0S1dKqrSIGjql7PoERNrZ6ZzZ028WktQxPXUrxo4hcGHzTxgcHHTXxs8H4T79cyN/GhweMmHht80sQnBp838fl/LLCUp/ENVYi7LogUiLkI3sw2fD8+KcOHZaxtZ0Mzv0EnGu0ZtGcUMOhYozcGvTFPFshu9T1TzQekWq9qDWtF24aGpm1Drepu27BrKqN+KI1sLY7dzPVBE9dJ3W/iep6dJlbHCT6k9dcyen/j5aDbf9b94afB6vPd6pP6gfWNtWKtW33re+u5dWCNrXPLsX6xfrV+s37v/NH5q/N355/S9f69KuZLq3U9sv4FgkaBxA==</latexit>

Original: [Table] [Table]
Revised: [Table] [Table]

Original: <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro
Revised: <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro

Original: [Table]
Revised: [Table]

Original: <latexit sha1_base64="+NT24K9PwZN8qZNmldQVB1jjcHA=">AAAMRHicjZbdbts2FMfV7quN95FuGHaxGyJBgBRIPDtLl/ViQBM7TQIEgZcmbdG5CyiZkohIpEJSsR1BT7Pb7TX2DnuH3Q27HXYoiZSUuMME2CJ/5xx+/M+hJDeJqFS93h/37r/3/gcffvTg4VLn408+/Wz50ecvJU+FR849HnHx2sWSRJSRc0VVRF4nguDYjcgr93Kg7a+uiZCUszM1T8jbGAeM+tTDCtDF8ldjRWZKetlBvHm6P8zX+5seF+TxxfJqr/u0v9Pf2UK97vZ3vW+/34bGk+2drSfbqN/tFdeqU12ji0edL8cT7qUxYcqLsJQ/9XuJepthoagXkXxpnEqSYO8SByRzOb9U2JU5Kq81lAjuE6mXiaPNqxRHVM0RuEREtiJxLH3OVCPSjcDkciwmKMYqRHIeuzy6E6WxCYKoaywYVyFlQcuRUY/4Anv18B6PwajMsMjnAvW/2dpARHndVmxMPcG1xrmJNYQHAifhvC1BnD+Eaw1RBZv1EAw+aTl4VJG88AgETxME/SJn7Z0VI1NvlmtHnwapuCWY1kRxWHnpAatf0VXDyNXVSlujKOCCqhAWnYOn7crFXvltgTUrlqtCAhV0K46wNIYNxcUqFEchviYoIkoRgbSNiAWbi9NIUcGndTYgMiBMO5NF1QEojbAANf6XexjqY3NrJ2I+kWHEioVOOCxwgrSXRJQhXI7SCpgVhzBfWlpbW0MjwbkPkxgN1DwiWRJhynJd5LqBfqhSvoEmxKeM6n0DFDzGbGnMyLQKzuBMJFwW9jwbNTotr3qQPBvW7ZZPdc+zs6rRskYkjnGeHRe3lgUeBTwCQed5NrDNlgcc9DROytl363Z7FNg2zD0obqVOPNFJ4QKkgvXrH8vGG2NTmeONvDRgEcQUTLqKeVLepQ9JCsYbYMjzhh+evcMPz6xfROM7TsDADumBBlWl36JJy/lQ5bBgtnKiykGmyR0HYHYllPl37MCsHZI/swbdsVvgQb10Hhh8Q4TF0DbYp/Uo0DZ4In2Lx1CIhg/Z4OQES5VlxXshK/r5z2NAeZU6D06pwvAILiOOtRLZ8UU9nKJsjkotqpDWI65IlxvDDPCH7X5dg1yLPIM8i4hBxKLQoNAiaZC0aGZQXQuBQYFFc4PmFt0YdGORMEhYdGXQlUXMoLpKU4NSi44NOrbo2qBri6YGTS3iMQmsjGXP2uDgqdpW9PJGnXBN4dczbNf47lq3oUFDi44MOlqUWDSFtwKCOiGCzqzmZSkVspsiMkJb07xtummYbtqld2e6ECs7FbQhaEonRLdMqqtIiVlVr2dQorZWz+zmXjTxi1qGJq6leN7Ezy0+bOJDi0+a+MTigyY+qGVu4iOLR008svi0iU8tPm/i8/9YYClP4xuqEHddEiUR9xG8mV34enxchg/KWNfNBnZ+i04N2rdo3ypg0YlBbyx6Y58skN3qe6aaD0i1Xt0a1Iq2DQ1N24Za1b22Yc9WRv1QGrpGHLeZ68MmrpN60MT1PLtNrI8TfEibr2X07sbLrW5/u/v0x63VZ3vVJ/UD52tnxVl3+s6O88w5dEbOueM5ufOL86vzW+f3zp+dvzp/l67371UxXzitq/PPv4OOgco=</latexit>
Revised: <latexit sha1_base64="+NT24K9PwZN8qZNmldQVB1jjcHA=">AAAMRHicjZbdbts2FMfV7quN95FuGHaxGyJBgBRIPDtLl/ViQBM7TQIEgZcmbdG5CyiZkohIpEJSsR1BT7Pb7TX2DnuH3Q27HXYoiZSUuMME2CJ/5xx+/M+hJDeJqFS93h/37r/3/gcffvTg4VLn408+/Wz50ecvJU+FR849HnHx2sWSRJSRc0VVRF4nguDYjcgr93Kg7a+uiZCUszM1T8jbGAeM+tTDCtDF8ldjRWZKetlBvHm6P8zX+5seF+TxxfJqr/u0v9Pf2UK97vZ3vW+/34bGk+2drSfbqN/tFdeqU12ji0edL8cT7qUxYcqLsJQ/9XuJepthoagXkXxpnEqSYO8SByRzOb9U2JU5Kq81lAjuE6mXiaPNqxRHVM0RuEREtiJxLH3OVCPSjcDkciwmKMYqRHIeuzy6E6WxCYKoaywYVyFlQcuRUY/4Anv18B6PwajMsMjnAvW/2dpARHndVmxMPcG1xrmJNYQHAifhvC1BnD+Eaw1RBZv1EAw+aTl4VJG88AgETxME/SJn7Z0VI1NvlmtHnwapuCWY1kRxWHnpAatf0VXDyNXVSlujKOCCqhAWnYOn7crFXvltgTUrlqtCAhV0K46wNIYNxcUqFEchviYoIkoRgbSNiAWbi9NIUcGndTYgMiBMO5NF1QEojbAANf6XexjqY3NrJ2I+kWHEioVOOCxwgrSXRJQhXI7SCpgVhzBfWlpbW0MjwbkPkxgN1DwiWRJhynJd5LqBfqhSvoEmxKeM6n0DFDzGbGnMyLQKzuBMJFwW9jwbNTotr3qQPBvW7ZZPdc+zs6rRskYkjnGeHRe3lgUeBTwCQed5NrDNlgcc9DROytl363Z7FNg2zD0obqVOPNFJ4QKkgvXrH8vGG2NTmeONvDRgEcQUTLqKeVLepQ9JCsYbYMjzhh+evcMPz6xfROM7TsDADumBBlWl36JJy/lQ5bBgtnKiykGmyR0HYHYllPl37MCsHZI/swbdsVvgQb10Hhh8Q4TF0DbYp/Uo0DZ4In2Lx1CIhg/Z4OQES5VlxXshK/r5z2NAeZU6D06pwvAILiOOtRLZ8UU9nKJsjkotqpDWI65IlxvDDPCH7X5dg1yLPIM8i4hBxKLQoNAiaZC0aGZQXQuBQYFFc4PmFt0YdGORMEhYdGXQlUXMoLpKU4NSi44NOrbo2qBri6YGTS3iMQmsjGXP2uDgqdpW9PJGnXBN4dczbNf47lq3oUFDi44MOlqUWDSFtwKCOiGCzqzmZSkVspsiMkJb07xtummYbtqld2e6ECs7FbQhaEonRLdMqqtIiVlVr2dQorZWz+zmXjTxi1qGJq6leN7Ezy0+bOJDi0+a+MTigyY+qGVu4iOLR008svi0iU8tPm/i8/9YYClP4xuqEHddEiUR9xG8mV34enxchg/KWNfNBnZ+i04N2rdo3ypg0YlBbyx6Y58skN3qe6aaD0i1Xt0a1Iq2DQ1N24Za1b22Yc9WRv1QGrpGHLeZ68MmrpN60MT1PLtNrI8TfEibr2X07sbLrW5/u/v0x63VZ3vVJ/UD52tnxVl3+s6O88w5dEbOueM5ufOL86vzW+f3zp+dvzp/l67371UxXzitq/PPv4OOgco=</latexit>

Original: A s yn c - R E DS G ( 8 - c o r e ) <latexit sha1_base64="drPGWQjJG3c2q59Rml2hbWhNeJo=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJAqRAktnZR5OLAU3sNAkQBF4+2qJ1F1AyJRGRSIWkYjuCrvY0u91eYy+w19jdsJsdSiIlJe4wAbbI3zmHH/9zKMlNIipVt/vnvfsfPPjwo48fPlroPP7k088Wn3z+UvJUeOTc4xEXr10sSUQZOVdUReR1IgiO3Yi8ci/72v7qmghJOTtTs4S8i3HAqE89rABdLH49UmSqpJftyBnz1k/2Buun+/nq1rrHBXl6sbjc3Xj27fdb29uou9EtLt3o/dDrbqFeRZad6hpePOl8ORpzL40JU16EpXzb6ybqXYaFol5E8oVRKkmCvUsckMzl/FJhV+aovFZQIrhPpF4sjtavUhxRNUPgEhHZisSx9DlTjUg3ApPLsRijGKsQyVns8uhOlMYmCKKusWBchZQFLUdGPeIL7NXDezwGozLDIp8L1Ptmcw0R5W20YmPqCa6Vzk2sITwQOAlnbQni/BFcK4gq2KyHYPBxy8GjiuSFRyB4miDoF5lr76wYmXrTXDv6NEjFLcG0JorDyksPWP2Srh1Grq6W2hpFARdUhbDoHDxtV873ym8LrFmxXBUSqKBbcYSlMWwoLlahOArxNUERUYoIpG1EzNlcnEaKCj6pswGRAWHamcyrDkBphAWo8b/cw1Afnls7EbOxDCNWLHTMYYFjpL0kogzhcpRWwLQ4ivnCwsrKChoKzn2YxGigZhHJkghTlusi1w30Y5XyNTQmPmVU7xug4DFmCyNGJlVwBmci4bKw59mw0Wl51YPk2aBut3yqe56dVY2WNSJxjPPsqLi1LPAo4BEIOsuzvm22POCgp3FSzr5Tt9ujwLZh7n5xK3XiiU4KFyAVrF//WDZaG5nKHK3lpQGLIKZg0lXMk/IufUhSMFoDQ543/PD0PX54av0iGt9xAgZ2SA80qCr95k1azocqhzmzlRNVDjJN7jgAsyuhzL9jB2btkPypNeiO3QIP6qXzwOAbIiyGtsE+rUeBtsFj6Vs8gkI0fMD6x8dYqiwr3g5Z0c9/HgHKq9R5cEoVhkdwGXGklciOLurhFGUzVGpRhbQecUW63BhmgD9s9+sa5FrkGeRZRAwiFoUGhRZJg6RFU4PqWggMCiyaGTSz6MagG4uEQcKiK4OuLGIG1VWaGpRadGTQkUXXBl1bNDFoYhGPSWBlLHvWBgdP1bailzfqhGsKv65hO8Z3x7oNDBpYdGjQ4bzEogm8FRDUCRF0ajUvS6mQ3RSREdqaZm3TTcN00y69O9OFWNmpoA1BEzomumVSXUVKzKp6PYMStbV6Zjd32sSntQxNXEvxoolfWHzQxAcWHzfxscX7Tbxfy9zEhxYPm3ho8UkTn1h83sTn/7HAUp7GN1Qh7qokSiLuI3gzu/AN+bQM75exrpv17fwWnRi0Z9GeVcCiY4PeWPTGPlkgu9X3TDUfkGq9utWvFW0bGpq2DbWqu23Drq2M+qE0cI04bjPXB01cJ3W/iet5dppYHyf4kDZfy+j9jZebG73vNrZ/2lx+vlt9Uj90vnKWnFWn5zxznjsHztA5dzznF+dX5zfn984fnb86f3f+KV3v36tivnBa1+MH/wKEa4MJ</latexit>
Revised: A s yn c - R E DS G ( 8 - c o r e ) <latexit sha1_base64="drPGWQjJG3c2q59Rml2hbWhNeJo=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJAqRAktnZR5OLAU3sNAkQBF4+2qJ1F1AyJRGRSIWkYjuCrvY0u91eYy+w19jdsJsdSiIlJe4wAbbI3zmHH/9zKMlNIipVt/vnvfsfPPjwo48fPlroPP7k088Wn3z+UvJUeOTc4xEXr10sSUQZOVdUReR1IgiO3Yi8ci/72v7qmghJOTtTs4S8i3HAqE89rABdLH49UmSqpJftyBnz1k/2Buun+/nq1rrHBXl6sbjc3Xj27fdb29uou9EtLt3o/dDrbqFeRZad6hpePOl8ORpzL40JU16EpXzb6ybqXYaFol5E8oVRKkmCvUsckMzl/FJhV+aovFZQIrhPpF4sjtavUhxRNUPgEhHZisSx9DlTjUg3ApPLsRijGKsQyVns8uhOlMYmCKKusWBchZQFLUdGPeIL7NXDezwGozLDIp8L1Ptmcw0R5W20YmPqCa6Vzk2sITwQOAlnbQni/BFcK4gq2KyHYPBxy8GjiuSFRyB4miDoF5lr76wYmXrTXDv6NEjFLcG0JorDyksPWP2Srh1Grq6W2hpFARdUhbDoHDxtV873ym8LrFmxXBUSqKBbcYSlMWwoLlahOArxNUERUYoIpG1EzNlcnEaKCj6pswGRAWHamcyrDkBphAWo8b/cw1Afnls7EbOxDCNWLHTMYYFjpL0kogzhcpRWwLQ4ivnCwsrKChoKzn2YxGigZhHJkghTlusi1w30Y5XyNTQmPmVU7xug4DFmCyNGJlVwBmci4bKw59mw0Wl51YPk2aBut3yqe56dVY2WNSJxjPPsqLi1LPAo4BEIOsuzvm22POCgp3FSzr5Tt9ujwLZh7n5xK3XiiU4KFyAVrF//WDZaG5nKHK3lpQGLIKZg0lXMk/IufUhSMFoDQ543/PD0PX54av0iGt9xAgZ2SA80qCr95k1azocqhzmzlRNVDjJN7jgAsyuhzL9jB2btkPypNeiO3QIP6qXzwOAbIiyGtsE+rUeBtsFj6Vs8gkI0fMD6x8dYqiwr3g5Z0c9/HgHKq9R5cEoVhkdwGXGklciOLurhFGUzVGpRhbQecUW63BhmgD9s9+sa5FrkGeRZRAwiFoUGhRZJg6RFU4PqWggMCiyaGTSz6MagG4uEQcKiK4OuLGIG1VWaGpRadGTQkUXXBl1bNDFoYhGPSWBlLHvWBgdP1bailzfqhGsKv65hO8Z3x7oNDBpYdGjQ4bzEogm8FRDUCRF0ajUvS6mQ3RSREdqaZm3TTcN00y69O9OFWNmpoA1BEzomumVSXUVKzKp6PYMStbV6Zjd32sSntQxNXEvxoolfWHzQxAcWHzfxscX7Tbxfy9zEhxYPm3ho8UkTn1h83sTn/7HAUp7GN1Qh7qokSiLuI3gzu/AN+bQM75exrpv17fwWnRi0Z9GeVcCiY4PeWPTGPlkgu9X3TDUfkGq9utWvFW0bGpq2DbWqu23Drq2M+qE0cI04bjPXB01cJ3W/iet5dppYHyf4kDZfy+j9jZebG73vNrZ/2lx+vlt9Uj90vnKWnFWn5zxznjsHztA5dzznF+dX5zfn984fnb86f3f+KV3v36tivnBa1+MH/wKEa4MJ</latexit>

Original: <latexit sha1_base64="fHNDNu5Lb9HcEwv7idQ0VJyIHT8=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJArRA4tlp0DQXA5rYaRIgCLw0aYvWXUDJlExEIhWSiq0IutrT7HZ7jb3AXmN3w252KImUlKTDBNgif+ccfvzPoSQ3DqlUvd6fd+5+cu/Tzz6//2Ch8/CLL79afPT1a8kT4ZFTj4dcvHWxJCFl5FRRFZK3sSA4ckPyxj0faPubSyIk5exEpTH5EOGAUZ96WAE6W/x+rMhcSS/blinz1o53h2s7e/nj52seF+TJ2eJyr/v86eaz/jPU625s9Te3NqHR7z3d2FhH/W6vuJad6hqdPep8O55wL4kIU16IpXzf78XqQ4aFol5I8oVxIkmMvXMckMzl/FxhV+aovFZQLLhPpF4sDtcuEhxSlSJwCYlsReJI+pypRqQbgsnlWExQhNUUyTRyeXgjSmMTBFGXWDCuppQFLUdGPeIL7NXDezwCozLDIp8L1P9hfRUR5XVbsRH1BNdK5ybWEB4IHE/TtgRR/gCuFUQVbNZDMPik5eBRRfLCIxA8iRH0i8y1d1aMTL15rh19GiTimmBaE8Vh5aUHrH5J1w4jFxdLbY3CgAuqprDoHDxtV97ulV8XWLNiuWpKoIKuxRGWRLChqFiF4miKLwkKiVJEIG0j4pbNRUmoqOCzOhsQGRCmnclt1QEoCbEANf6X+3SqD8+1nYh0IqchKxY64bDACdJeElGGcDlKK2BeHMV8YWFlZQWNBOc+TGI0UGlIsjjElOW6yHUD/VilfBVNiE8Z1fsGKHiE2cKYkVkVnMGZiLks7Hk2anRaXvUgeTas2y2f6p5nJ1WjZQ1JFOE8OyxuLQs8CngIgqZ5NrDNlgcc9CSKy9m363Z7FNg2zD0obqVOPNZJ4QKkgvXrH8vGq2NTmePVvDRgEUQUTLqKeVzepQ9JCsarYMjzhh+ef8QPz61fSKMbTsDADumBBlWl322TlvOhyuGW2cqJKgeZxDccgNmVUObfsAOzdkj+3Bp0x26BB/XSeWDwFREWQ9tgn9ajQNvgifQtHkMhGj5kg6MjLFWWFW+HrOjnP48B5VXqPDilCsMjuIw41Epkh2f1cIqyFJVaVCGtR1yRLjeCGeAP2/26BrkWeQZ5FhGDiEVTg6YWSYOkRXOD6loIDAosSg1KLboy6MoiYZCw6MKgC4uYQXWVJgYlFh0adGjRpUGXFs0MmlnEIxJYGcuetcHBU7Wt6OWNOuGawq9n2Lbx3bZuQ4OGFh0YdHBbYtEM3goI6oQIOreal6VUyG6KyAhtTWnbdNUwXbVL78Z0U6zsVNCGoBmdEN0yqa4iJWZVvZ5AidpaPbGbe9XEr2oZmriW4mUTv7R4v4n3LT5q4iOL95p4r5a5iQ8sHjXxyOLjJj62+LSJT/9jgaU8jW+oQtzHkiiJuI/gzezCN+STMnxQxrpuNrDzW3Rs0K5Fu1YBi44MemfRO/tkgexW3zPVfECq9erWoFa0bWho2jbUqu60DTu2MuqH0tA14rjNXO83cZ3UvSau59luYn2c4EPafC2jjzder3f7G92tn9aXX+xUn9T3ne+cJeex03c2nRfOvjNyTh3P+cX51fnN+b3zR+evzt+df0rXu3eqmG+c1vXw3r/gGIMR</latexit>
Revised: <latexit sha1_base64="fHNDNu5Lb9HcEwv7idQ0VJyIHT8=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJArRA4tlp0DQXA5rYaRIgCLw0aYvWXUDJlExEIhWSiq0IutrT7HZ7jb3AXmN3w252KImUlKTDBNgif+ccfvzPoSQ3DqlUvd6fd+5+cu/Tzz6//2Ch8/CLL79afPT1a8kT4ZFTj4dcvHWxJCFl5FRRFZK3sSA4ckPyxj0faPubSyIk5exEpTH5EOGAUZ96WAE6W/x+rMhcSS/blinz1o53h2s7e/nj52seF+TJ2eJyr/v86eaz/jPU625s9Te3NqHR7z3d2FhH/W6vuJad6hqdPep8O55wL4kIU16IpXzf78XqQ4aFol5I8oVxIkmMvXMckMzl/FxhV+aovFZQLLhPpF4sDtcuEhxSlSJwCYlsReJI+pypRqQbgsnlWExQhNUUyTRyeXgjSmMTBFGXWDCuppQFLUdGPeIL7NXDezwCozLDIp8L1P9hfRUR5XVbsRH1BNdK5ybWEB4IHE/TtgRR/gCuFUQVbNZDMPik5eBRRfLCIxA8iRH0i8y1d1aMTL15rh19GiTimmBaE8Vh5aUHrH5J1w4jFxdLbY3CgAuqprDoHDxtV97ulV8XWLNiuWpKoIKuxRGWRLChqFiF4miKLwkKiVJEIG0j4pbNRUmoqOCzOhsQGRCmnclt1QEoCbEANf6X+3SqD8+1nYh0IqchKxY64bDACdJeElGGcDlKK2BeHMV8YWFlZQWNBOc+TGI0UGlIsjjElOW6yHUD/VilfBVNiE8Z1fsGKHiE2cKYkVkVnMGZiLks7Hk2anRaXvUgeTas2y2f6p5nJ1WjZQ1JFOE8OyxuLQs8CngIgqZ5NrDNlgcc9CSKy9m363Z7FNg2zD0obqVOPNZJ4QKkgvXrH8vGq2NTmePVvDRgEUQUTLqKeVzepQ9JCsarYMjzhh+ef8QPz61fSKMbTsDADumBBlWl322TlvOhyuGW2cqJKgeZxDccgNmVUObfsAOzdkj+3Bp0x26BB/XSeWDwFREWQ9tgn9ajQNvgifQtHkMhGj5kg6MjLFWWFW+HrOjnP48B5VXqPDilCsMjuIw41Epkh2f1cIqyFJVaVCGtR1yRLjeCGeAP2/26BrkWeQZ5FhGDiEVTg6YWSYOkRXOD6loIDAosSg1KLboy6MoiYZCw6MKgC4uYQXWVJgYlFh0adGjRpUGXFs0MmlnEIxJYGcuetcHBU7Wt6OWNOuGawq9n2Lbx3bZuQ4OGFh0YdHBbYtEM3goI6oQIOreal6VUyG6KyAhtTWnbdNUwXbVL78Z0U6zsVNCGoBmdEN0yqa4iJWZVvZ5AidpaPbGbe9XEr2oZmriW4mUTv7R4v4n3LT5q4iOL95p4r5a5iQ8sHjXxyOLjJj62+LSJT/9jgaU8jW+oQtzHkiiJuI/gzezCN+STMnxQxrpuNrDzW3Rs0K5Fu1YBi44MemfRO/tkgexW3zPVfECq9erWoFa0bWho2jbUqu60DTu2MuqH0tA14rjNXO83cZ3UvSau59luYn2c4EPafC2jjzder3f7G92tn9aXX+xUn9T3ne+cJeex03c2nRfOvjNyTh3P+cX51fnN+b3zR+evzt+df0rXu3eqmG+c1vXw3r/gGIMR</latexit>

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: and variance of the stochastic gradient (Lian et al., 2015; Ghadimi & Lan, 2016).

Original: Our ﬁnal assumption is related to the deep denoiser used in A SYNC -RED.

Original: Assumption 4.

Original: The denoiser D  is a nonexpansive operator k D  ( x )  D  ( y ) k  k x  y k .

Original: Compared with the conditions stated in Section 2 (namely, that it is locally homogeneous with a symmetric Jacobian), our requirement on the denoiser is milder.
Revised: Compared with the conditions stated in Section 2 (namely, that it is locally homogeneous with a symmetric Jacobian), our requirement on the denoiser is milder.

Original: One can train a nonexpansive D  by constraining the Lipschitz constant of D  via the spectral normalization, which is an active area of research in deep learning (Miyato et al., 2018; Sedghi et al., 2019; Anil et al., 2019).
Revised: One can train a nonexpansive D  by constraining the Lipschitz constant of D  via the spectral normalization, which is an active area of research in deep learning (Miyato et al., 2018; Sedghi et al., 2019; Anil et al., 2019; Terris et al., 2020).

Original: We can now state the theorems on A SYNC -RED.
Revised: We can now state the theorems on A SYNC -RED.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let Assumptions 1-4 hold true.
Revised: Let Assumptions 1-4 hold true.

Original: Run A SYNC -RED -BG for t > 0 iterations with uniform i.i.d block selection using a ﬁxed step-size  2 (0 , 1 / ((1 + 2  )
Revised: Run A SYNC -RED -BG for t > 0 iterations with uniform i.i.d block selection using a ﬁxed step-size  2 (0 , 1 / ((1 + 2  )

Original: ( L + 2 ⌧ ))] .
Revised: ( L + 2 ⌧ ))] .

Original: Then, the iterates of the algorithm satisfy
Revised: Then, the iterates of the algorithm satisfy

Original: [Equation]
Revised: [Equation]

Original: where D = 2  2 / (1 +  ) 2 is a constant.
Revised: where D = 2  2 / (1 +  ) 2 is a constant.

Original: Theorem 1 establishes the convergence of A SYNC -RED-BG to the ﬁxed-point set zer ( G ) at the rate of O (1 /t ) .
Revised: Theorem 1 establishes the convergence of A SYNC -RED-BG to the ﬁxed-point set zer ( G ) at the rate of O (1 /t ) .

Original: Our result is consistent with the existing results in the literature.
Revised: Our result is consistent with the existing results in the literature.

Original: In particular, when the algorithm adopts serial block updates, that is  = 0 and e x k = x k , the recovered convergence is nearly the same as BC-RED (Sun et al., 2019a) scaled by some constant.
Revised: In particular, when the algorithm adopts serial block updates, that is  = 0 and e x k = x k , the recovered convergence is nearly the same as BC-RED (Sun et al., 2019a) scaled by some constant.

Original: On the other hand, our convergence rate O (1 /t ) is also consistent with the rate proved for the asynchronous block coordinate descent in nonconvex optimization (Sun et al., 2017).
Revised: On the other hand, our convergence rate O (1 /t ) is also consistent with the rate proved for the asynchronous block coordinate descent in nonconvex optimization (Sun et al., 2017).

Original: Theorem 2.
Revised: Theorem 2.

Original: Let Assumptions 1-4 hold true.
Revised: Let Assumptions 1-4 hold true.

Original: Run A SYNC -RED -SG for t > 0 iterations with uniform i.i.d selections of blocks and measurements using a ﬁxed step-size  2 (0 , 1 / ((1 + 2  )( L + 2 ⌧ ))] .
Revised: Run A SYNC -RED -SG for t > 0 iterations with uniform i.i.d selections of blocks and measurements using a ﬁxed step-size  2 (0 , 1 / ((1 + 2  )( L + 2 ⌧ ))] .

Original: Then, the iterates of the algorithm satisfy
Revised: Then, the iterates of the algorithm satisfy

Original: [Equation]
Revised: [Equation]

Original: where C =
Revised: where C =

Original: ( L + 2 ⌧ )(1 +  ) ⌫ 2 and D = 2  2 / (1 +  ) 2 are constants.
Revised: ( L + 2 ⌧ )(1 +  ) ⌫ 2 and D = 2  2 / (1 +  ) 2 are constants.

Original: Theorem 2 states that A SYNC -RED-SG approximates the solution obtained by A SYNC -RED-BG up to a ﬁnite error that decreases for larger values of the minibatch size w .
Revised: Theorem 2 states that A SYNC -RED-SG approximates the solution obtained by A SYNC -RED-BG up to a ﬁnite error that decreases for larger values of the minibatch size w .

Original: This relationship is consistent with the recent theoretical results on the online PnP and RED algorithms (Sun et al., 2019b; Wu et al., 2020).
Revised: This relationship is consistent with the recent theoretical results on the online PnP and RED algorithms (Sun et al., 2019b; Wu et al., 2020).

Original: In practice, the selection of w must balance the actual memory capacity of the system and the desired runtime for obtaining a reasonable solution.
Revised: In practice, the selection of w must balance the actual memory capacity of the system and the desired runtime for obtaining a reasonable solution.

Original: Our numerical evaluation in Sectiondemonstrates the excellent approximation of A SYNC -RED-SG to the batch-gradient solution by using a small subset of data.
Revised: Our numerical evaluation in Sectiondemonstrates the excellent approximation of A SYNC -RED-SG to the batch-gradient solution by using a small subset of data.

Original: By carefully choosing the stepsize  , we can state the following remark on Theorem 2.
Revised: By carefully choosing the stepsize  , we can state the following remark on Theorem 2.

Original: Remark 1.
Revised: Remark 1.

Original: Set the stepsize to be  = 1 / p wt .
Revised: Set the stepsize to be  = 1 / p wt .

Original: If the maximal delay satisﬁes   (1 / 2)
Revised: If the maximal delay satisﬁes   (1 / 2)

Original: [ p wt/ ( L + 2 ⌧ )  1] , then after t > 0 iterations we have
Revised: [ p wt/ ( L + 2 ⌧ )  1] , then after t > 0 iterations we have

Original: [Equation]
Revised: [Equation]

Revised: This establishes the ﬁxed-point convergence to the set zer ( G ) at the rate of O (1 / p wt ) under speciﬁc conditions.

Revised: If we treat entire x as a block, namely that U

Revised: = U T

Revised: =

Revised: I and b = 1 , A SYNC -RED-SG then becomes the asynchronous stochastic RED algorithm.

Revised: Hence, the proposed remark immediately

Original: S N R ( d
Revised: S N R ( d

Original: B ) <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro+ONWBzTIj8pbx2PJ6SIQNBFkQ/R7ETAQc/itJp9r7G7o8C2Ye5heat0EqkpipAgFazf/CX5ZHtiO3OyXVQOKoOYg8t0sUiru/KhSMFkGxxF0Yqj83+Jo3OMi3i8FAQM/FAeMLiu4lZNWs1H6oAVs1UT1QEqS5cCgOFKeOIv+YGhH4o/R4d5wC2IoFm6CCy+YxIx2Bb7vBkFbIunykc8gUa0/CAZjkZU6TyfaDbXeflcvJsAKurSeXBKNYVXcJVxYpTIT66a4TRPFqTSok7pvOLKcrkxzAD/KO7XtchF5FnkIWIWMUShRSEiZZFCNLeo6YXAogDRwqIFojuL7hBJiySiG4tuECUWNV2aWZQhOrHoBNGtRbeIZhbNEImYBShj9YQ+OHi68ZVPRatPhKHw17dsz8buYdiBRQeIji06XlVYMoNfBQJ9wiSfo+ZVK5Wy2yayQqNr0XXdtVx33dZbmi6kGqcCG5JmfMqMZUtdZyqa1P16Di2KvXqOmztr47NGhjZupHjZxi8RH7XxEeJRG48QH7bxYSNzGx8jHrfxGPFpG58ivmjji/9YYCVP6xuqFHdLMa2I8An8MrvwsfisSh9Wua6bD3F+RKcWvUD0AhVANLLoDaI3+GaB6tbfM/V8QOr1GmvYKNp1tDTtOhpV97uOfeyM5qV04Fpx3Hatj9q4KephGzfz7LWxOU7wIT24/9m8bLza3Rl8u/P9j7sbz/frT+pHzlfOurPlDJzvnOfOkTN2LhzPiZyfnV+cX3u/9f7o/dn7qwp9+KDO+cLpXL2//wGxJXq7</latexit>
Revised: B ) <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro+ONWBzTIj8pbx2PJ6SIQNBFkQ/R7ETAQc/itJp9r7G7o8C2Ye5heat0EqkpipAgFazf/CX5ZHtiO3OyXVQOKoOYg8t0sUiru/KhSMFkGxxF0Yqj83+Jo3OMi3i8FAQM/FAeMLiu4lZNWs1H6oAVs1UT1QEqS5cCgOFKeOIv+YGhH4o/R4d5wC2IoFm6CCy+YxIx2Bb7vBkFbIunykc8gUa0/CAZjkZU6TyfaDbXeflcvJsAKurSeXBKNYVXcJVxYpTIT66a4TRPFqTSok7pvOLKcrkxzAD/KO7XtchF5FnkIWIWMUShRSEiZZFCNLeo6YXAogDRwqIFojuL7hBJiySiG4tuECUWNV2aWZQhOrHoBNGtRbeIZhbNEImYBShj9YQ+OHi68ZVPRatPhKHw17dsz8buYdiBRQeIji06XlVYMoNfBQJ9wiSfo+ZVK5Wy2yayQqNr0XXdtVx33dZbmi6kGqcCG5JmfMqMZUtdZyqa1P16Di2KvXqOmztr47NGhjZupHjZxi8RH7XxEeJRG48QH7bxYSNzGx8jHrfxGPFpG58ivmjji/9YYCVP6xuqFHdLMa2I8An8MrvwsfisSh9Wua6bD3F+RKcWvUD0AhVANLLoDaI3+GaB6tbfM/V8QOr1GmvYKNp1tDTtOhpV97uOfeyM5qV04Fpx3Hatj9q4KephGzfz7LWxOU7wIT24/9m8bLza3Rl8u/P9j7sbz/frT+pHzlfOurPlDJzvnOfOkTN2LhzPiZyfnV+cX3u/9f7o/dn7qwp9+KDO+cLpXL2//wGxJXq7</latexit>

Original: [Figure]
Revised: [Figure]

Original: latexit sha1_base64="9ZZ/bPBwBFHpWjncVOEQW4JPq7g
Revised: latexit sha1_base64="9ZZ/bPBwBFHpWjncVOEQW4JPq7g

Original: [Table] [Table]
Revised: [Table] [Table]

Original: O784v/Z+6/3R+7P3V2V6/17t86XTeXp//wOfJH5I</latexit> iteration ( k )
Revised: O784v/Z+6/3R+7P3V2V6/17t86XTeXp//wOfJH5I</latexit> iteration ( k )

Original: ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR
Revised: ) k <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR

Original: /2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU/Xix86yN+iW6XNvob/fLiywag9rYcOprdPlk5Vt3LPwsZon2I6rUL4N+qj/kVGruR6xYdjPFUupf0ZDlnhBXmnqqINXVI6kUAVNm0TT6/jqjEddzAiERU51MGqtAJLqV6UXg8gSVYxJTPSFqHnsiWsgy2CZB1g2VidATnoSdwIT7LJDUb4b3RQxObYclgZBk8GxnizDtb3dyY+5LYRQvbK4lIpQ0ncy7EsTFY7h6hGvYrE9g8HEnwOeaFWVEKEWWEnguK9jdWTky92eFCQx4mMk7ghlNtICVVxGw+nXTQwm7vl7vahSFQnI9gUUXEImP6v6o4q7AhpXL1RMmJLuTx5Ishg3F5Sq0IBN6w0jEtGaSGB
Revised: /2hknFRXKm5yn7ENMw4QH3qQZ0udZzP7oHKth0Z158cfXU/Xix86yN+iW6XNvob/fLiywag9rYcOprdPlk5Vt3LPwsZon2I6rUL4N+qj/kVGruR6xYdjPFUupf0ZDlnhBXmnqqINXVI6kUAVNm0TT6/jqjEddzAiERU51MGqtAJLqV6UXg8gSVYxJTPSFqHnsiWsgy2CZB1g2VidATnoSdwIT7LJDUb4b3RQxObYclgZBk8GxnizDtb3dyY+5LYRQvbK4lIpQ0ncy7EsTFY7h6hGvYrE9g8HEnwOeaFWVEKEWWEnguK9jdWTky92eFCQx4mMk7ghlNtICVVxGw+nXTQwm7vl7vahSFQnI9gUUXEImP6v6o4q7AhpXL1RMmJLuTx5Ishg3F5Sq0IBN6w0jEtGaSGB

Original: [Table] [Table]
Revised: [Table] [Table]

Original: RED
Revised: RED

Original: [Table]
Revised: [Table]

Original: ( n c =
Revised: ( n c =

Original: [Table]
Revised: [Table]

Original: /fj5vKLneqb+r7zlbPkrDo956nzwtl3hs6p4zm/OL86vzm/d/7o/NX5u/NP6Xr3ThXzhdO6Ht77F/ZCgw4=</latexit> Async-RED-BG ( n c =8) <
Revised: /fj5vKLneqb+r7zlbPkrDo956nzwtl3hs6p4zm/OL86vzm/d/7o/NX5u/NP6Xr3ThXzhdO6Ht77F/ZCgw4=</latexit> Async-RED-BG ( n c =8) <

Original: [Table]
Revised: [Table]

Original: +JYTMLBDeqBBVem3aNJyPlQ5LJitnKhykGlyywGYXQll/i07MGuH5M+sQXfsFnhQL50HBl8TYTG0DfZpPQq0DZ5I3+IxFKLhQzY4OsJSZVnxYsiKfv7zGFBepc6DU6owPILLiEOtRHZ4Xg+nKJujUosqpPWIK9LlxjAD/GG7X9cg1yLPIM8iYhCxKDQotEgaJC2aGVTXQmBQYNHcoLlF1wZdWyQMEhZdGnRpETOortLUoNSiQ4MOLboy6MqiqUFTi3hMAitj2bM2OHiqthW9vFEnXFP49QzbNr7b1m1o0NCiA4MOFiUWTeGtgKBOiKAzq3lZSoXspoiM0NY0b5uuG6brdundmi7Eyk4FbQia0gnRLZPqKlJiVtXrCZSordUTu7lXTfyqlqGJayleNvFLi/ebeN/ioyY+snivifdqmZv4wOJRE48sPm7iY4tPm/j0PxZYytP4hirEfSyJkoj7CN7MLnw+PinDB2Ws62YDO79FxwbtWrRrFbDoyKA3Fr2xTxbIbvU9U80HpFqvbg1qRduGhqZtQ63qTtuwYyujfigNXSOO28z1fhPXSd1r4nqe7SbWxwk+pM3XMnp342yj29/s9vs/bKy+2Km+qe85XzkrzmOn72w5L5x9Z+ScOp7zi/Or85vze+ePzl+dvzv/lK5371Qxnzut64HzL4zlgLI=</latexit> BC-RED ( n c =1) 8000 10101010 8000 time (second)
Revised: +JYTMLBDeqBBVem3aNJyPlQ5LJitnKhykGlyywGYXQll/i07MGuH5M+sQXfsFnhQL50HBl8TYTG0DfZpPQq0DZ5I3+IxFKLhQzY4OsJSZVnxYsiKfv7zGFBepc6DU6owPILLiEOtRHZ4Xg+nKJujUosqpPWIK9LlxjAD/GG7X9cg1yLPIM8iYhCxKDQotEgaJC2aGVTXQmBQYNHcoLlF1wZdWyQMEhZdGnRpETOortLUoNSiQ4MOLboy6MqiqUFTi3hMAitj2bM2OHiqthW9vFEnXFP49QzbNr7b1m1o0NCiA4MOFiUWTeGtgKBOiKAzq3lZSoXspoiM0NY0b5uuG6brdundmi7Eyk4FbQia0gnRLZPqKlJiVtXrCZSordUTu7lXTfyqlqGJayleNvFLi/ebeN/ioyY+snivifdqmZv4wOJRE48sPm7iY4tPm/j0PxZYytP4hirEfSyJkoj7CN7MLnw+PinDB2Ws62YDO79FxwbtWrRrFbDoyKA3Fr2xTxbIbvU9U80HpFqvbg1qRduGhqZtQ63qTtuwYyujfigNXSOO28z1fhPXSd1r4nqe7SbWxwk+pM3XMnp342yj29/s9vs/bKy+2Km+qe85XzkrzmOn72w5L5x9Z+ScOp7zi/Or85vze+ePzl+dvzv/lK5371Qxnzut64HzL4zlgLI=</latexit> BC-RED ( n c =1) 8000 10101010 8000 time (second)

Original: G
Revised: G

Original: <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv
Revised: <latexit sha1_base64="E74wdyBuHn/ONfvGmuD10zG1IFM=">AAAMUnicjZbdTtxGFMdN0o8EaEta9ao3I9BKRKJkF0WqelEpsCSAhNCWQBKlDmjsHXtH2B4zM2Z3cfwEfZretq/Rm75Kr3rG9hzbLK1qCXz8O+fMx/+c8dpLI650v//X0oOHn3z62eePHi+vrH7x5VdrT75+o0QmfXbui0jIdx5VLOIJO9dcR+xdKhmNvYi99a6Gxv

Original: [Table]
Revised: [Table]

Original: /MPRD8WfoMA+4BRE2SxehxbdMIgbb4oA3o4Bt8VgFiF1oRMv3k+HJCVU6z13NZjovn4sLF1BRl86HU6opvIKrjGOjRH582QyneTInlRZ1SucVV5bLi2EG+Edxv55FHiLfIh8Rs4ghmlg0QaQsUohmFjW9EFoUIppbNEd0a9EtImmRRHRt0TWixKKmSzOLMkTHFh0jurHoBtHUoikiEbMQZaye0AcHTze+8qlo9YkwFP76lu3a2F0M27doH9GRRUf3FZZM4VeBQJ8wyWeoedVKpey2iazQ6Jp3Xbct12239Ramm1CNU4ENSVM+Zsaypa4zFU3qfj2DFsVePcPNvW7j140MbdxI8aqNXyE+bONDxCdtfIL4oI0PGpnb+AjxqI1HiE/b+BTxeRuf/8cCK3la31CluJuKaUVEQOCX2YNvyadV+rDK9bx8iPMjOrXoJaKXqACiE4veI3qPbxaobv09U88HpF6vsYaNol1HS9Ouo1F1r+vYw85oXkr7nhXHa9f6sI2boh60cTPPbhub4wQf0oO7n82Lxpud7cHz7R9/fr7xYq/+pH7kfOesO5vOwPnBeeEcOiPn3PGdX53fnN+dP1b+XPl7dWn1YRX6YKnO+cbpXKur/wBgkYPv</latexit>
Revised: /MPRD8WfoMA+4BRE2SxehxbdMIgbb4oA3o4Bt8VgFiF1oRMv3k+HJCVU6z13NZjovn4sLF1BRl86HU6opvIKrjGOjRH582QyneTInlRZ1SucVV5bLi2EG+Edxv55FHiLfIh8Rs4ghmlg0QaQsUohmFjW9EFoUIppbNEd0a9EtImmRRHRt0TWixKKmSzOLMkTHFh0jurHoBtHUoikiEbMQZaye0AcHTze+8qlo9YkwFP76lu3a2F0M27doH9GRRUf3FZZM4VeBQJ8wyWeoedVKpey2iazQ6Jp3Xbct12239Ramm1CNU4ENSVM+Zsaypa4zFU3qfj2DFsVePcPNvW7j140MbdxI8aqNXyE+bONDxCdtfIL4oI0PGpnb+AjxqI1HiE/b+BTxeRuf/8cCK3la31CluJuKaUVEQOCX2YNvyadV+rDK9bx8iPMjOrXoJaKXqACiE4veI3qPbxaobv09U88HpF6vsYaNol1HS9Ouo1F1r+vYw85oXkr7nhXHa9f6sI2boh60cTPPbhub4wQf0oO7n82Lxpud7cHz7R9/fr7xYq/+pH7kfOesO5vOwPnBeeEcOiPn3PGdX53fnN+dP1b+XPl7dWn1YRX6YKnO+cbpXKur/wBgkYPv</latexit>

Original: <latexit sha1_base64="J3bg65h54GDAwZq87E+RuqOxJ3A=">AAAMRnicjZbdbts2FMfV7quNuy3dsIthN0SCACmQerbTZdnFgCZ2mgQIAi8fbdG6CyiZkohIokJSsRVB2NPsdnuNvcJeYnfDbncoiZSUuMME2CJ/5xx+/M+hJDsOqJC93p/37n/w4Ucff/Lg4VLn0aeffb78+IuXgiXcIecOCxh/bWNBAhqRc0llQF7HnODQDsgr+3Ko7K+uCReURWcyjcm7EHsRdamDJaCL5a8nksylcLLTNHKenuyN8vXtpw7j5MnF8mqvu7W9vflsgHrdwdbWZl81eptbg+/6qN/tFdeqVV3ji8edryZT5iQhiaQTYCHe9nuxfJdhLqkTkHxpkggSY+cSeySzGbuU2BY5Kq81FHPmEqEWioOnVwkOqEwRuAREtCJxKFwWyUakHYDJZphPUYilj0Qa2iy4E6WwDoKoa8wjJn0aeS3HiDrE5diph3dYCEaph0Uu46j/7WADEel0W7EhdThTKuc6VhPmcRz7aVuCMH8I1xqiEjbrIBh82nJwqCR54eFxlsQI+kXW2jsrRqbOPFeOLvUSfkswpYlksPLSA1a/ouomIldXK22NAo9xKn1YdA6episWe+W3BVasWK70CVTQrTgSJSFsKCxWIRny8TVBAZGScKRshC/YXJgEknI2q7MBkR6JlDNZVB2AkgBzUON/ufu+Oji3dsLTqfCDqFjolMECp0h5CUQjhMtRWgHz4hjmS0tra2tozBlzYRKtgUwDksUBplGuilw10I9VyjfQlLg0omrfADkLcbQ0icisCs7gTMRMFPY8Gzc6La96kDwb1e2WT3XPs7Oq0bIGJAxxnh0Vt5YFHgUsAEHTPBuaZssDDnoSxuXsO3W7PQpsG+YeFrdSJxarpDAOUsH61S/KJhsTXZmTjbw0YO6FFEyqillc3oULSfImG2DI84Yfnr/HD8+NX0DDO07AwA7pgQaVpd+iScv5UOWwYLZyospBJPEdB2BmJTRy79iBGTskf24MqmO2wLx66czT+IZwg6GtsUvrUaCt8VS4Bk+gEDUfRcPjYyxklhVvhqzo5z9PAOVV6hw4pRLDI7iMOFJKZEcX9XCSRikqtahCWo+4Il12CDPAHzb7tTWyDXI0cgwiGhGDfI18g4RGwqC5RnUteBp5BqUapQbdaHRjENeIG3Sl0ZVBkUZ1lSYaJQYdaXRk0LVG1wbNNJoZxELiGRnLnrHBwZO1rejljTphisKvp9mO9t0xbiONRgYdanS4KLFoBm8FBHVCOJ0bzctSKmTXRaSFNqa0bbppmG7apXdnOh9LMxW0IWhGp0S1dKqrSIGjql7PoERNrZ6ZzZ028WktQxPXUrxo4hcGHzTxgcHHTXxs8H4T79cyN/GhweMmHht80sQnBp838fl/LLCUp/ENVYi7LogUiLkI3sw2fD8+KcOHZaxtZ0Mzv0EnGu0ZtGcUMOhYozcGvTFPFshu9T1TzQekWq9qDWtF24aGpm1Drepu27BrKqN+KI1sLY7dzPVBE9dJ3W/iep6dJlbHCT6k9dcyen/j5aDbf9b94afB6vPd6pP6gfWNtWKtW33re+u5dWCNrXPLsX6xfrV+s37v/NH5q/N355/S9f69KuZLq3U9sv4FgkaBxA==</latexit>
Revised: <latexit sha1_base64="J3bg65h54GDAwZq87E+RuqOxJ3A=">AAAMRnicjZbdbts2FMfV7quNuy3dsIthN0SCACmQerbTZdnFgCZ2mgQIAi8fbdG6CyiZkohIokJSsRVB2NPsdnuNvcJeYnfDbncoiZSUuMME2CJ/5xx+/M+hJDsOqJC93p/37n/w4Ucff/Lg4VLn0aeffb78+IuXgiXcIecOCxh/bWNBAhqRc0llQF7HnODQDsgr+3Ko7K+uCReURWcyjcm7EHsRdamDJaCL5a8nksylcLLTNHKenuyN8vXtpw7j5MnF8mqvu7W9vflsgHrdwdbWZl81eptbg+/6qN/tFdeqVV3ji8edryZT5iQhiaQTYCHe9nuxfJdhLqkTkHxpkggSY+cSeySzGbuU2BY5Kq81FHPmEqEWioOnVwkOqEwRuAREtCJxKFwWyUakHYDJZphPUYilj0Qa2iy4E6WwDoKoa8wjJn0aeS3HiDrE5diph3dYCEaph0Uu46j/7WADEel0W7EhdThTKuc6VhPmcRz7aVuCMH8I1xqiEjbrIBh82nJwqCR54eFxlsQI+kXW2jsrRqbOPFeOLvUSfkswpYlksPLSA1a/ouomIldXK22NAo9xKn1YdA6episWe+W3BVasWK70CVTQrTgSJSFsKCxWIRny8TVBAZGScKRshC/YXJgEknI2q7MBkR6JlDNZVB2AkgBzUON/ufu+Oji3dsLTqfCDqFjolMECp0h5CUQjhMtRWgHz4hjmS0tra2tozBlzYRKtgUwDksUBplGuilw10I9VyjfQlLg0omrfADkLcbQ0icisCs7gTMRMFPY8Gzc6La96kDwb1e2WT3XPs7Oq0bIGJAxxnh0Vt5YFHgUsAEHTPBuaZssDDnoSxuXsO3W7PQpsG+YeFrdSJxarpDAOUsH61S/KJhsTXZmTjbw0YO6FFEyqillc3oULSfImG2DI84Yfnr/HD8+NX0DDO07AwA7pgQaVpd+iScv5UOWwYLZyospBJPEdB2BmJTRy79iBGTskf24MqmO2wLx66czT+IZwg6GtsUvrUaCt8VS4Bk+gEDUfRcPjYyxklhVvhqzo5z9PAOVV6hw4pRLDI7iMOFJKZEcX9XCSRikqtahCWo+4Il12CDPAHzb7tTWyDXI0cgwiGhGDfI18g4RGwqC5RnUteBp5BqUapQbdaHRjENeIG3Sl0ZVBkUZ1lSYaJQYdaXRk0LVG1wbNNJoZxELiGRnLnrHBwZO1rejljTphisKvp9mO9t0xbiONRgYdanS4KLFoBm8FBHVCOJ0bzctSKmTXRaSFNqa0bbppmG7apXdnOh9LMxW0IWhGp0S1dKqrSIGjql7PoERNrZ6ZzZ028WktQxPXUrxo4hcGHzTxgcHHTXxs8H4T79cyN/GhweMmHht80sQnBp838fl/LLCUp/ENVYi7LogUiLkI3sw2fD8+KcOHZaxtZ0Mzv0EnGu0ZtGcUMOhYozcGvTFPFshu9T1TzQekWq9qDWtF24aGpm1Drepu27BrKqN+KI1sLY7dzPVBE9dJ3W/iep6dJlbHCT6k9dcyen/j5aDbf9b94afB6vPd6pP6gfWNtWKtW33re+u5dWCNrXPLsX6xfrV+s37v/NH5q/N355/S9f69KuZLq3U9sv4FgkaBxA==</latexit>

Original: [Figure]
Revised: [Figure]

Original: <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro
Revised: <latexit sha1_base64="S35mgHp8mCs1Yp8Xmy+UQ9HA2bI=">AAAMM3icjZbbbttGEIaZ9JRYPSQtetWbhQ0DDuC6klGg6EWB2HJiGzAE1YfESJUYS2pJLkxy6d2lJZngS/S2fY0+TNG7ord9h86S3CFpqUUJ2Bx+M7OHf2YpumnEle73f3/w8L33P/jwo0eP13off/LpZ0+efv5KiUx67MITkZCXLlUs4gm70FxH7DKVjMZuxF6710Pjf33LpOIiOdeLlL2NaZBwn3tUA7o8G52Sren+s6snG/2dfnmRZWNQGxtOfY2vnva+nEyFl8Us0V5Elfpp0E/125xKzb2IFWuTTLGUetc0YLkrxLWmripIdW2SVAqfKbMqGn19k9GI6wWBkIipTiaNlS8S3cp0I3C5gsopiakOiVrEroiWsgy2SZB1S2UidMiToBOYcI/5knrN8J6IwantsMQXkgy+2d0mTHs7ndyYe1IYSQuba4kIJE3DRVeCuHgM1ybhGjbrERh82gnwuGZFGRFIkaUEnssSdXdWjsy9eWECfR5k8p5gRhMtYOVVBKx+3TRJwm5u1rsaRYGQXIew6AIi8VGtjiruC2xYuVwdMiHZvTyWZDFsKC5XoQUJ6S0jEdOaSWJ8TK7YXJxFmksxa6oBmQFLTDBb1R2AsohKUON/hYehOSX3diIXUxVGSbnQqYAFTomJUoQnhFajdBLm5Zkr1tY2NzfJWArhwyRWA72IWJ5GlCeFaXJjkB/qkm+TKfN5ws2+AUoR02RtkrBZnZzDmUiFKv1FPm49dKKaQYr8oLE7MfW9yM9ro

Original: [Table]
Revised: [Table]

Original: <latexit sha1_base64="+NT24K9PwZN8qZNmldQVB1jjcHA=">AAAMRHicjZbdbts2FMfV7quN95FuGHaxGyJBgBRIPDtLl/ViQBM7TQIEgZcmbdG5CyiZkohIpEJSsR1BT7Pb7TX2DnuH3Q27HXYoiZSUuMME2CJ/5xx+/M+hJDeJqFS93h/37r/3/gcffvTg4VLn408+/Wz50ecvJU+FR849HnHx2sWSRJSRc0VVRF4nguDYjcgr93Kg7a+uiZCUszM1T8jbGAeM+tTDCtDF8ldjRWZKetlBvHm6P8zX+5seF+TxxfJqr/u0v9Pf2UK97vZ3vW+/34bGk+2drSfbqN/tFdeqU12ji0edL8cT7qUxYcqLsJQ/9XuJepthoagXkXxpnEqSYO8SByRzOb9U2JU5Kq81lAjuE6mXiaPNqxRHVM0RuEREtiJxLH3OVCPSjcDkciwmKMYqRHIeuzy6E6WxCYKoaywYVyFlQcuRUY/4Anv18B6PwajMsMjnAvW/2dpARHndVmxMPcG1xrmJNYQHAifhvC1BnD+Eaw1RBZv1EAw+aTl4VJG88AgETxME/SJn7Z0VI1NvlmtHnwapuCWY1kRxWHnpAatf0VXDyNXVSlujKOCCqhAWnYOn7crFXvltgTUrlqtCAhV0K46wNIYNxcUqFEchviYoIkoRgbSNiAWbi9NIUcGndTYgMiBMO5NF1QEojbAANf6XexjqY3NrJ2I+kWHEioVOOCxwgrSXRJQhXI7SCpgVhzBfWlpbW0MjwbkPkxgN1DwiWRJhynJd5LqBfqhSvoEmxKeM6n0DFDzGbGnMyLQKzuBMJFwW9jwbNTotr3qQPBvW7ZZPdc+zs6rRskYkjnGeHRe3lgUeBTwCQed5NrDNlgcc9DROytl363Z7FNg2zD0obqVOPNFJ4QKkgvXrH8vGG2NTmeONvDRgEcQUTLqKeVLepQ9JCsYbYMjzhh+evcMPz6xfROM7TsDADumBBlWl36JJy/lQ5bBgtnKiykGmyR0HYHYllPl37MCsHZI/swbdsVvgQb10Hhh8Q4TF0DbYp/Uo0DZ4In2Lx1CIhg/Z4OQES5VlxXshK/r5z2NAeZU6D06pwvAILiOOtRLZ8UU9nKJsjkotqpDWI65IlxvDDPCH7X5dg1yLPIM8i4hBxKLQoNAiaZC0aGZQXQuBQYFFc4PmFt0YdGORMEhYdGXQlUXMoLpKU4NSi44NOrbo2qBri6YGTS3iMQmsjGXP2uDgqdpW9PJGnXBN4dczbNf47lq3oUFDi44MOlqUWDSFtwKCOiGCzqzmZSkVspsiMkJb07xtummYbtqld2e6ECs7FbQhaEonRLdMqqtIiVlVr2dQorZWz+zmXjTxi1qGJq6leN7Ezy0+bOJDi0+a+MTigyY+qGVu4iOLR008svi0iU8tPm/i8/9YYClP4xuqEHddEiUR9xG8mV34enxchg/KWNfNBnZ+i04N2rdo3ypg0YlBbyx6Y58skN3qe6aaD0i1Xt0a1Iq2DQ1N24Za1b22Yc9WRv1QGrpGHLeZ68MmrpN60MT1PLtNrI8TfEibr2X07sbLrW5/u/v0x63VZ3vVJ/UD52tnxVl3+s6O88w5dEbOueM5ufOL86vzW+f3zp+dvzp/l67371UxXzitq/PPv4OOgco=</latexit>
Revised: <latexit sha1_base64="+NT24K9PwZN8qZNmldQVB1jjcHA=">AAAMRHicjZbdbts2FMfV7quN95FuGHaxGyJBgBRIPDtLl/ViQBM7TQIEgZcmbdG5CyiZkohIpEJSsR1BT7Pb7TX2DnuH3Q27HXYoiZSUuMME2CJ/5xx+/M+hJDeJqFS93h/37r/3/gcffvTg4VLn408+/Wz50ecvJU+FR849HnHx2sWSRJSRc0VVRF4nguDYjcgr93Kg7a+uiZCUszM1T8jbGAeM+tTDCtDF8ldjRWZKetlBvHm6P8zX+5seF+TxxfJqr/u0v9Pf2UK97vZ3vW+/34bGk+2drSfbqN/tFdeqU12ji0edL8cT7qUxYcqLsJQ/9XuJepthoagXkXxpnEqSYO8SByRzOb9U2JU5Kq81lAjuE6mXiaPNqxRHVM0RuEREtiJxLH3OVCPSjcDkciwmKMYqRHIeuzy6E6WxCYKoaywYVyFlQcuRUY/4Anv18B6PwajMsMjnAvW/2dpARHndVmxMPcG1xrmJNYQHAifhvC1BnD+Eaw1RBZv1EAw+aTl4VJG88AgETxME/SJn7Z0VI1NvlmtHnwapuCWY1kRxWHnpAatf0VXDyNXVSlujKOCCqhAWnYOn7crFXvltgTUrlqtCAhV0K46wNIYNxcUqFEchviYoIkoRgbSNiAWbi9NIUcGndTYgMiBMO5NF1QEojbAANf6XexjqY3NrJ2I+kWHEioVOOCxwgrSXRJQhXI7SCpgVhzBfWlpbW0MjwbkPkxgN1DwiWRJhynJd5LqBfqhSvoEmxKeM6n0DFDzGbGnMyLQKzuBMJFwW9jwbNTotr3qQPBvW7ZZPdc+zs6rRskYkjnGeHRe3lgUeBTwCQed5NrDNlgcc9DROytl363Z7FNg2zD0obqVOPNFJ4QKkgvXrH8vGG2NTmeONvDRgEcQUTLqKeVLepQ9JCsYbYMjzhh+evcMPz6xfROM7TsDADumBBlWl36JJy/lQ5bBgtnKiykGmyR0HYHYllPl37MCsHZI/swbdsVvgQb10Hhh8Q4TF0DbYp/Uo0DZ4In2Lx1CIhg/Z4OQES5VlxXshK/r5z2NAeZU6D06pwvAILiOOtRLZ8UU9nKJsjkotqpDWI65IlxvDDPCH7X5dg1yLPIM8i4hBxKLQoNAiaZC0aGZQXQuBQYFFc4PmFt0YdGORMEhYdGXQlUXMoLpKU4NSi44NOrbo2qBri6YGTS3iMQmsjGXP2uDgqdpW9PJGnXBN4dczbNf47lq3oUFDi44MOlqUWDSFtwKCOiGCzqzmZSkVspsiMkJb07xtummYbtqld2e6ECs7FbQhaEonRLdMqqtIiVlVr2dQorZWz+zmXjTxi1qGJq6leN7Ezy0+bOJDi0+a+MTigyY+qGVu4iOLR008svi0iU8tPm/i8/9YYClP4xuqEHddEiUR9xG8mV34enxchg/KWNfNBnZ+i04N2rdo3ypg0YlBbyx6Y58skN3qe6aaD0i1Xt0a1Iq2DQ1N24Za1b22Yc9WRv1QGrpGHLeZ68MmrpN60MT1PLtNrI8TfEibr2X07sbLrW5/u/v0x63VZ3vVJ/UD52tnxVl3+s6O88w5dEbOueM5ufOL86vzW+f3zp+dvzp/l67371UxXzitq/PPv4OOgco=</latexit>

Original: <latexit sha1_base64="drPGWQjJG3c2q59Rml2hbWhNeJo=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJAqRAktnZR5OLAU3sNAkQBF4+2qJ1F1AyJRGRSIWkYjuCrvY0u91eYy+w19jdsJsdSiIlJe4wAbbI3zmHH/9zKMlNIipVt/vnvfsfPPjwo48fPlroPP7k088Wn3z+UvJUeOTc4xEXr10sSUQZOVdUReR1IgiO3Yi8ci/72v7qmghJOTtTs4S8i3HAqE89rABdLH49UmSqpJftyBnz1k/2Buun+/nq1rrHBXl6sbjc3Xj27fdb29uou9EtLt3o/dDrbqFeRZad6hpePOl8ORpzL40JU16EpXzb6ybqXYaFol5E8oVRKkmCvUsckMzl/FJhV+aovFZQIrhPpF4sjtavUhxRNUPgEhHZisSx9DlTjUg3ApPLsRijGKsQyVns8uhOlMYmCKKusWBchZQFLUdGPeIL7NXDezwGozLDIp8L1Ptmcw0R5W20YmPqCa6Vzk2sITwQOAlnbQni/BFcK4gq2KyHYPBxy8GjiuSFRyB4miDoF5lr76wYmXrTXDv6NEjFLcG0JorDyksPWP2Srh1Grq6W2hpFARdUhbDoHDxtV873ym8LrFmxXBUSqKBbcYSlMWwoLlahOArxNUERUYoIpG1EzNlcnEaKCj6pswGRAWHamcyrDkBphAWo8b/cw1Afnls7EbOxDCNWLHTMYYFjpL0kogzhcpRWwLQ4ivnCwsrKChoKzn2YxGigZhHJkghTlusi1w30Y5XyNTQmPmVU7xug4DFmCyNGJlVwBmci4bKw59mw0Wl51YPk2aBut3yqe56dVY2WNSJxjPPsqLi1LPAo4BEIOsuzvm22POCgp3FSzr5Tt9ujwLZh7n5xK3XiiU4KFyAVrF//WDZaG5nKHK3lpQGLIKZg0lXMk/IufUhSMFoDQ543/PD0PX54av0iGt9xAgZ2SA80qCr95k1azocqhzmzlRNVDjJN7jgAsyuhzL9jB2btkPypNeiO3QIP6qXzwOAbIiyGtsE+rUeBtsFj6Vs8gkI0fMD6x8dYqiwr3g5Z0c9/HgHKq9R5cEoVhkdwGXGklciOLurhFGUzVGpRhbQecUW63BhmgD9s9+sa5FrkGeRZRAwiFoUGhRZJg6RFU4PqWggMCiyaGTSz6MagG4uEQcKiK4OuLGIG1VWaGpRadGTQkUXXBl1bNDFoYhGPSWBlLHvWBgdP1bailzfqhGsKv65hO8Z3x7oNDBpYdGjQ4bzEogm8FRDUCRF0ajUvS6mQ3RSREdqaZm3TTcN00y69O9OFWNmpoA1BEzomumVSXUVKzKp6PYMStbV6Zjd32sSntQxNXEvxoolfWHzQxAcWHzfxscX7Tbxfy9zEhxYPm3ho8UkTn1h83sTn/7HAUp7GN1Qh7qokSiLuI3gzu/AN+bQM75exrpv17fwWnRi0Z9GeVcCiY4PeWPTGPlkgu9X3TDUfkGq9utWvFW0bGpq2DbWqu23Drq2M+qE0cI04bjPXB01cJ3W/iet5dppYHyf4kDZfy+j9jZebG73vNrZ/2lx+vlt9Uj90vnKWnFWn5zxznjsHztA5dzznF+dX5zfn984fnb86f3f+KV3v36tivnBa1+MH/wKEa4MJ</latexit>
Revised: <latexit sha1_base64="drPGWQjJG3c2q59Rml2hbWhNeJo=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJAqRAktnZR5OLAU3sNAkQBF4+2qJ1F1AyJRGRSIWkYjuCrvY0u91eYy+w19jdsJsdSiIlJe4wAbbI3zmHH/9zKMlNIipVt/vnvfsfPPjwo48fPlroPP7k088Wn3z+UvJUeOTc4xEXr10sSUQZOVdUReR1IgiO3Yi8ci/72v7qmghJOTtTs4S8i3HAqE89rABdLH49UmSqpJftyBnz1k/2Buun+/nq1rrHBXl6sbjc3Xj27fdb29uou9EtLt3o/dDrbqFeRZad6hpePOl8ORpzL40JU16EpXzb6ybqXYaFol5E8oVRKkmCvUsckMzl/FJhV+aovFZQIrhPpF4sjtavUhxRNUPgEhHZisSx9DlTjUg3ApPLsRijGKsQyVns8uhOlMYmCKKusWBchZQFLUdGPeIL7NXDezwGozLDIp8L1Ptmcw0R5W20YmPqCa6Vzk2sITwQOAlnbQni/BFcK4gq2KyHYPBxy8GjiuSFRyB4miDoF5lr76wYmXrTXDv6NEjFLcG0JorDyksPWP2Srh1Grq6W2hpFARdUhbDoHDxtV873ym8LrFmxXBUSqKBbcYSlMWwoLlahOArxNUERUYoIpG1EzNlcnEaKCj6pswGRAWHamcyrDkBphAWo8b/cw1Afnls7EbOxDCNWLHTMYYFjpL0kogzhcpRWwLQ4ivnCwsrKChoKzn2YxGigZhHJkghTlusi1w30Y5XyNTQmPmVU7xug4DFmCyNGJlVwBmci4bKw59mw0Wl51YPk2aBut3yqe56dVY2WNSJxjPPsqLi1LPAo4BEIOsuzvm22POCgp3FSzr5Tt9ujwLZh7n5xK3XiiU4KFyAVrF//WDZaG5nKHK3lpQGLIKZg0lXMk/IufUhSMFoDQ543/PD0PX54av0iGt9xAgZ2SA80qCr95k1azocqhzmzlRNVDjJN7jgAsyuhzL9jB2btkPypNeiO3QIP6qXzwOAbIiyGtsE+rUeBtsFj6Vs8gkI0fMD6x8dYqiwr3g5Z0c9/HgHKq9R5cEoVhkdwGXGklciOLurhFGUzVGpRhbQecUW63BhmgD9s9+sa5FrkGeRZRAwiFoUGhRZJg6RFU4PqWggMCiyaGTSz6MagG4uEQcKiK4OuLGIG1VWaGpRadGTQkUXXBl1bNDFoYhGPSWBlLHvWBgdP1bailzfqhGsKv65hO8Z3x7oNDBpYdGjQ4bzEogm8FRDUCRF0ajUvS6mQ3RSREdqaZm3TTcN00y69O9OFWNmpoA1BEzomumVSXUVKzKp6PYMStbV6Zjd32sSntQxNXEvxoolfWHzQxAcWHzfxscX7Tbxfy9zEhxYPm3ho8UkTn1h83sTn/7HAUp7GN1Qh7qokSiLuI3gzu/AN+bQM75exrpv17fwWnRi0Z9GeVcCiY4PeWPTGPlkgu9X3TDUfkGq9utWvFW0bGpq2DbWqu23Drq2M+qE0cI04bjPXB01cJ3W/iet5dppYHyf4kDZfy+j9jZebG73vNrZ/2lx+vlt9Uj90vnKWnFWn5zxznjsHztA5dzznF+dX5zfn984fnb86f3f+KV3v36tivnBa1+MH/wKEa4MJ</latexit>

Original: <latexit sha1_base64="fHNDNu5Lb9HcEwv7idQ0VJyIHT8=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJArRA4tlp0DQXA5rYaRIgCLw0aYvWXUDJlExEIhWSiq0IutrT7HZ7jb3AXmN3w252KImUlKTDBNgif+ccfvzPoSQ3DqlUvd6fd+5+cu/Tzz6//2Ch8/CLL79afPT1a8kT4ZFTj4dcvHWxJCFl5FRRFZK3sSA4ckPyxj0faPubSyIk5exEpTH5EOGAUZ96WAE6W/x+rMhcSS/blinz1o53h2s7e/nj52seF+TJ2eJyr/v86eaz/jPU625s9Te3NqHR7z3d2FhH/W6vuJad6hqdPep8O55wL4kIU16IpXzf78XqQ4aFol5I8oVxIkmMvXMckMzl/FxhV+aovFZQLLhPpF4sDtcuEhxSlSJwCYlsReJI+pypRqQbgsnlWExQhNUUyTRyeXgjSmMTBFGXWDCuppQFLUdGPeIL7NXDezwCozLDIp8L1P9hfRUR5XVbsRH1BNdK5ybWEB4IHE/TtgRR/gCuFUQVbNZDMPik5eBRRfLCIxA8iRH0i8y1d1aMTL15rh19GiTimmBaE8Vh5aUHrH5J1w4jFxdLbY3CgAuqprDoHDxtV97ulV8XWLNiuWpKoIKuxRGWRLChqFiF4miKLwkKiVJEIG0j4pbNRUmoqOCzOhsQGRCmnclt1QEoCbEANf6X+3SqD8+1nYh0IqchKxY64bDACdJeElGGcDlKK2BeHMV8YWFlZQWNBOc+TGI0UGlIsjjElOW6yHUD/VilfBVNiE8Z1fsGKHiE2cKYkVkVnMGZiLks7Hk2anRaXvUgeTas2y2f6p5nJ1WjZQ1JFOE8OyxuLQs8CngIgqZ5NrDNlgcc9CSKy9m363Z7FNg2zD0obqVOPNZJ4QKkgvXrH8vGq2NTmePVvDRgEUQUTLqKeVzepQ9JCsarYMjzhh+ef8QPz61fSKMbTsDADumBBlWl322TlvOhyuGW2cqJKgeZxDccgNmVUObfsAOzdkj+3Bp0x26BB/XSeWDwFREWQ9tgn9ajQNvgifQtHkMhGj5kg6MjLFWWFW+HrOjnP48B5VXqPDilCsMjuIw41Epkh2f1cIqyFJVaVCGtR1yRLjeCGeAP2/26BrkWeQZ5FhGDiEVTg6YWSYOkRXOD6loIDAosSg1KLboy6MoiYZCw6MKgC4uYQXWVJgYlFh0adGjRpUGXFs0MmlnEIxJYGcuetcHBU7Wt6OWNOuGawq9n2Lbx3bZuQ4OGFh0YdHBbYtEM3goI6oQIOreal6VUyG6KyAhtTWnbdNUwXbVL78Z0U6zsVNCGoBmdEN0yqa4iJWZVvZ5AidpaPbGbe9XEr2oZmriW4mUTv7R4v4n3LT5q4iOL95p4r5a5iQ8sHjXxyOLjJj62+LSJT/9jgaU8jW+oQtzHkiiJuI/gzezCN+STMnxQxrpuNrDzW3Rs0K5Fu1YBi44MemfRO/tkgexW3zPVfECq9erWoFa0bWho2jbUqu60DTu2MuqH0tA14rjNXO83cZ3UvSau59luYn2c4EPafC2jjzder3f7G92tn9aXX+xUn9T3ne+cJeex03c2nRfOvjNyTh3P+cX51fnN+b3zR+evzt+df0rXu3eqmG+c1vXw3r/gGIMR</latexit>
Revised: <latexit sha1_base64="fHNDNu5Lb9HcEwv7idQ0VJyIHT8=">AAAMSnicjZbdbts2FMfVrtvauNvSDbsaBhAJArRA4tlp0DQXA5rYaRIgCLw0aYvWXUDJlExEIhWSiq0IutrT7HZ7jb3AXmN3w252KImUlKTDBNgif+ccfvzPoSQ3DqlUvd6fd+5+cu/Tzz6//2Ch8/CLL79afPT1a8kT4ZFTj4dcvHWxJCFl5FRRFZK3sSA4ckPyxj0faPubSyIk5exEpTH5EOGAUZ96WAE6W/x+rMhcSS/blinz1o53h2s7e/nj52seF+TJ2eJyr/v86eaz/jPU625s9Te3NqHR7z3d2FhH/W6vuJad6hqdPep8O55wL4kIU16IpXzf78XqQ4aFol5I8oVxIkmMvXMckMzl/FxhV+aovFZQLLhPpF4sDtcuEhxSlSJwCYlsReJI+pypRqQbgsnlWExQhNUUyTRyeXgjSmMTBFGXWDCuppQFLUdGPeIL7NXDezwCozLDIp8L1P9hfRUR5XVbsRH1BNdK5ybWEB4IHE/TtgRR/gCuFUQVbNZDMPik5eBRRfLCIxA8iRH0i8y1d1aMTL15rh19GiTimmBaE8Vh5aUHrH5J1w4jFxdLbY3CgAuqprDoHDxtV97ulV8XWLNiuWpKoIKuxRGWRLChqFiF4miKLwkKiVJEIG0j4pbNRUmoqOCzOhsQGRCmnclt1QEoCbEANf6X+3SqD8+1nYh0IqchKxY64bDACdJeElGGcDlKK2BeHMV8YWFlZQWNBOc+TGI0UGlIsjjElOW6yHUD/VilfBVNiE8Z1fsGKHiE2cKYkVkVnMGZiLks7Hk2anRaXvUgeTas2y2f6p5nJ1WjZQ1JFOE8OyxuLQs8CngIgqZ5NrDNlgcc9CSKy9m363Z7FNg2zD0obqVOPNZJ4QKkgvXrH8vGq2NTmePVvDRgEUQUTLqKeVzepQ9JCsarYMjzhh+ef8QPz61fSKMbTsDADumBBlWl322TlvOhyuGW2cqJKgeZxDccgNmVUObfsAOzdkj+3Bp0x26BB/XSeWDwFREWQ9tgn9ajQNvgifQtHkMhGj5kg6MjLFWWFW+HrOjnP48B5VXqPDilCsMjuIw41Epkh2f1cIqyFJVaVCGtR1yRLjeCGeAP2/26BrkWeQZ5FhGDiEVTg6YWSYOkRXOD6loIDAosSg1KLboy6MoiYZCw6MKgC4uYQXWVJgYlFh0adGjRpUGXFs0MmlnEIxJYGcuetcHBU7Wt6OWNOuGawq9n2Lbx3bZuQ4OGFh0YdHBbYtEM3goI6oQIOreal6VUyG6KyAhtTWnbdNUwXbVL78Z0U6zsVNCGoBmdEN0yqa4iJWZVvZ5AidpaPbGbe9XEr2oZmriW4mUTv7R4v4n3LT5q4iOL95p4r5a5iQ8sHjXxyOLjJj62+LSJT/9jgaU8jW+oQtzHkiiJuI/gzezCN+STMnxQxrpuNrDzW3Rs0K5Fu1YBi44MemfRO/tkgexW3zPVfECq9erWoFa0bWho2jbUqu60DTu2MuqH0tA14rjNXO83cZ3UvSau59luYn2c4EPafC2jjzder3f7G92tn9aXX+xUn9T3ne+cJeex03c2nRfOvjNyTh3P+cX51fnN+b3zR+evzt+df0rXu3eqmG+c1vXw3r/gGIMR</latexit>

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: This establishes the ﬁxed-point convergence to the set zer ( G ) at the rate of O (1 / p wt ) under speciﬁc conditions.

Original: If we treat entire x as a block, namely that U

Original: = U T = I and b = 1 , A SYNC -RED-SG then becomes the asynchronous stochastic RED algorithm. Hence, the proposed remark immediately holds true for the later.
Revised: holds true for the later.

Original: Note that our convergence rate O
Revised: Note that our convergence rate O

Original: (1 / p wt ) is consistent with the rate proved for the serial (Nemirovski et al., 2009) and parallel (Dekel et al., 2012; Lian et al 2015) stochastic gradient methods.
Revised: (1 / p wt ) is consistent with the rate proved for the serial (Nemirovski et al., 2009) and parallel (Dekel et al., 2012; Lian et al., 2015) stochastic gradient methods.

Original: All the proofs are presented in the supplement.
Revised: All the proofs are presented in the supplement.

Original: Our analysis never assumes the existence of an explicit regularizer, and hence applicable to advanced denoisers that are not associated with any regularizer.
Revised: We note that the analysis above does not assume the existence of an explicit regularizer associated with the operator D  . Moreover, it does not require D  to be a Gaussian denoiser.

Revised: Our analysis is hence applicable to all nonexpansive operators, such as the traditional proximal operators or the more recent artifact-removal operators (Zhang et al., 2019).

Original:  N UMERICAL V ALIDATION
Revised:  N UMERICAL V ALIDATION

Original: We now present a numerical validation of A SYNC -RED.
Revised: We now present a numerical validation of A SYNC -RED.

Original: Our goals are ﬁrst to validate the proposed theorems in Section 4 and then to demonstrate the effectiveness and the efﬁciency of our algorithm on the large-scale problem.
Revised: Our goals are ﬁrst to validate the proposed theorems in Section 4 and then to demonstrate the effectiveness and the efﬁciency of our algorithm on the large-scale problem.

Original: We consider two image recovery tasks that have the form y
Revised: We consider two image recovery tasks that have the form y

Original: =
Revised: =

Original: Ax + e , where the measurement matrix
Revised: Ax + e , where the measurement matrix

Original: A corresponds to either the random matrix in compressive sensing (CS) or the Radon transform in computed tomography (CT) , and the noise e is assumed to be additive
Revised: A corresponds to either the random matrix in compressive sensing (CS) or the Radon transform in computed tomography (CT) , and the noise e is assumed to be additive

Original: white
Revised: white

Original: Gaussian (AWGN).
Revised: Gaussian (AWGN).

Original: In particular, the random matrix is implemented with the block-diagonal structure
Revised: In particular, the random matrix is implemented with the block-diagonal structure

Original: A = diag ([ A i , ..., A b ]) for fast validation, while the Radon transform is used as its full matrix form to demonstrate the effectiveness of A SYNC -RED for overcoming the computation bottleneck.
Revised: A = diag ([ A i , ..., A b ]) for fast validation, while the Radon transform is used as its full matrix form to demonstrate the effectiveness of A SYNC -RED for overcoming the computation bottleneck.

Original: Our deep neural net prior adapts the DnCNN architecture (Zhang et al., 2017a).
Revised: Our deep neural net prior adapts the DnCNN architecture (Zhang et al., 2017a).

Original: We used the signalto-noise ratio (dB) to quantify the quality of the reconstructed images.
Revised: We used the signalto-noise ratio (dB) to quantify the quality of the reconstructed images.

Original: For each experiments, we selected the denoiser that achieves the best SNR performance from the ones corresponding to ﬁve noise levels  2 { 5 , 10 , 15 , 20 , 25 } .
Revised: For each experiments, we selected the denoiser that achieves the best SNR performance from the ones corresponding to ﬁve noise levels  2 { 5 , 10 , 15 , 20 , 25 } .

Revised: The value of  is ﬁxed across all iterations of the algorithm.

Original: Supplement D provides additional technical details.
Revised: Supplement D provides additional technical details.

Original: 5. C ONVERGENCE B EHAVIOR
Revised: 5. C ONVERGENCE B EHAVIOR

Original: We validate our theorems on the CS task with 6 test images selected from the Set 12 dataset (Zhang et al., 2017a).
Revised: We validate our theorems on the CS task with 6 test images selected from the Set 12 dataset (Zhang et al., 2017a).

Original: Each test image is rescaled to the size of 240 ⇥ 240 pixels (see Fig.
Revised: Each test image is rescaled to the size of 240 ⇥ 240 pixels (see Fig.

Original: 6 in the supplement for the visualization).
Revised: 6 in the supplement for the visualization).

Original: The block-diagonal matrix A is set to consist of 9 submatrices, corresponding to a 3 ⇥ 3 grid of blocks with the size of 80 ⇥ 80 pixels in every image.
Revised: The block-diagonal matrix A is set to consist of 9 submatrices, corresponding to a 3 ⇥ 3 grid of blocks with the size of 80 ⇥ 80 pixels in every image.

Original: The elements in A are i.i.d zero-mean Gaussian random variables of variance of 1 /m , and the compression ratio is set to be m/n = 0 .
Revised: The elements in A are i.i.d zero-mean Gaussian random variables of variance of 1 /m , and the compression ratio is set to be m/n = 0 .

Original: 7 , which indicates that the total number of measurements is 4480 for each block.
Revised: 7 , which indicates that the total number of measurements is 4480 for each block.

Original: We obtain the measurements by multiplying A with each vectorized image and adding additional noise corresponding to the input SNR of 30 dB. Finally, we use the normalized distance k G ( x k ) k 22 / k G ( x 0 ) k 22 to quantify the ﬁxed-point convergence, with b block updates grouped as one iteration.
Revised: We obtain the measurements by multiplying A with each vectorized image and adding additional noise corresponding to the input SNR of 30 dB. Finally, we use the normalized distance k G ( x k ) k 22 / k G ( x 0 ) k 22 to quantify the ﬁxed-point convergence, with b block updates grouped as one iteration.

Original: The distance is expected to approach zero as the algorithm converges to a ﬁxed point.
Revised: The distance is expected to approach zero as the algorithm converges to a ﬁxed point.

Revised: The average performance of all methods is obtained by running a single trial for each image.

Original: Theorem 1 establishes the convergence of A SYNC -RED-BG to the ﬁxed point set zer ( G ) .
Revised: Theorem 1 establishes the convergence of A SYNC -RED-BG to the ﬁxed point set zer ( G ) .

Original: This is illustrated in Fig.
Revised: This is illustrated in Fig.

Original: 2 for four different numbers of accessible cores n c 2 { 2 , 4 , 6 , 8 } .
Revised: 2 for four different numbers of accessible cores n c 2 { 2 , 4 , 6 , 8 } .

Original: In the left ﬁgure, the average normalized distance is plotted against the iteration number, while the middle and right ﬁgures plot the corresponding distance and SNR values against the actual runtime in seconds.
Revised: In the left ﬁgure, the average normalized distance is plotted against the iteration number, while the middle and right ﬁgures plot the corresponding distance and SNR values against the actual runtime in seconds.

Original: The shaded areas representing the range of values attained across all test images.
Revised: The shaded areas representing the range of values attained across all images

Original: We alsoplot the results ofserial BC-RED using the dashed line as reference. A SYNC -RED-BG isimplemented to be run asynchronously on multiple cores, while BC-RED can only use one core to perform the
Revised: We also the results of is

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  computation.
Revised: to be run asynchronously on multiple cores, while BC-RED can only use one core to perform the computation.

Original: The left ﬁgure highlights the ﬁxed-point convergence of A SYNC -RED-BG in iteration for different n c , with all variants agreeing with the serial BC-RED.
Revised: The left ﬁgure highlights the ﬁxed-point convergence of A SYNC -RED-BG in iteration for different n c , with all variants agreeing with the serial BC-RED.

Original: Since A SYNC -RED-BG uses more cores, the middle and right ﬁgures demonstrate the signiﬁcantly faster in-time convergence of A SYNC -RED-BG than BC-RED to the same SNR value.
Revised: Since A SYNC -RED-BG uses more cores, the middle and right ﬁgures demonstrate the signiﬁcantly faster in-time convergence of A SYNC -RED-BG than BC-RED to the same SNR value.

Original: Speciﬁcally, BC-RED takes 1 .
Revised: Speciﬁcally, BC-RED takes 1 .

Original: 8 hours to achieve 29 . 

Revised: 8 hours to achieve 29 .

Original: dB, while A SYNC -RED-BG ( n c = 8 ) takes only 17 .
Revised: dB, while A SYNC -RED-BG ( n c = 8 ) takes only 17 .

Original: 9 minutes to obtain the same value, corresponding to a 6 ⇥ improvement in computation time.
Revised: 9 minutes to obtain the same value, corresponding to a 6 ⇥ improvement in computation time.

Original: Theorem 2 establishes the convergence of A SYNC -RED-SG to zer ( G ) up to some error term, which is inversely proportional to the minibatch size w .
Revised: Theorem 2 establishes the convergence of A SYNC -RED-SG to zer ( G ) up to some error term, which is inversely proportional to the minibatch size w .

Original: This is illustrated in Fig.
Revised: This is illustrated in Fig.

Original: 3 (left) for three different minibatch sizes w 2 { 1120 , 2240 , 3360 } .
Revised: 3 (left) for three different minibatch sizes w 2 { 1120 , 2240 , 3360 } .

Original: As before, we plotted the average distance against the iteration number with the shading area representing the variance.
Revised: As before, we plotted the average distance against the iteration number with the shading area representing the variance.

Original: Note that the log-scale of yaxis highlights the change for smaller values.
Revised: Note that the log-scale of yaxis highlights the change for smaller values.

Original: Fig.
Revised: Fig.

Original: 3 demonstrates the improved convergence of
Revised: 3 demonstrates the improved convergence of

Original: A SYNC -RED-SG to zer ( G ) for larger w , which is consistent with our theoretical analysis.
Revised: A SYNC -RED-SG to zer ( G ) for larger w , which is consistent with our theoretical analysis.

Original: Fig.(middle) compares the convergence speed between A SYNC -RED-BG/SG, gradient-method RED (G M -RED), and synchronous parallel RED (S YNC -RED).
Revised: Fig.(middle) compares the convergence speed between A SYNC -RED-BG/SG, gradient-method RED (G M -RED), and synchronous parallel RED (S YNC -RED).

Original: For A SYNC -RED-SG, we use w = 1120 .
Revised: For A SYNC -RED-SG, we use w = 1120 .

Original: In particular, A SYNC -RED-SG takes fewer total runtime (from 17.9 min to 13.0 min) to obtain the similar result ( 29 . 01 dB and 28 . 03 dB) and achieves 8 .
Revised: In particular, A SYNC -RED-SG takes fewer total runtime (from 17.9 min to 13.0 min) to obtain the similar result ( 29 . 01 dB and 28 . 03 dB) and achieves 8 .

Original: 4 ⇥ speedup compared with G M -RED.
Revised: 4 ⇥ speedup compared with G M -RED.

Original: The table in Fig.
Revised: The table in Fig.

Original: 3 summarizes the detailed results.
Revised: 3 summarizes the detailed results.

Original: 5. E FFECTIVENESS FOR C OMPUTATIONAL I MAGING
Revised: 5. E FFECTIVENESS FOR C OMPUTATIONAL I MAGING

Original: We additionally demonstrate the effectiveness of our algorithm by reconstructing a 800 ⇥ 800 CT image from its 180 projections.
Revised: We additionally demonstrate the effectiveness of our algorithm by reconstructing a 800 ⇥ 800 CT image from its 180 projections.

Original: For block parallel updates, the image is decomposed into 16 blocks, each having the size of 200 ⇥ 200 pixels.
Revised: For block parallel updates, the image is decomposed into 16 blocks, each having the size of 200 ⇥ 200 pixels.

Original: The Radon matrix used in the experiment corresponds to 180 angles with 1131 detectors, and the noise level is set to 70 dB. We refer to Supplement D. for additional technical details.
Revised: The Radon matrix used in the experiment corresponds to 180 angles with 1131 detectors, and the noise level is set to 70 dB. We refer to Supplement D. for additional technical details.

Original: Fig.
Revised: Fig.

Original: 4 shows the visual illustration of the reconstructed images by
Revised: 4 shows the visual illustration of the reconstructed images by

Original: A SYNC -RED-BG/SG and G M -RED.
Revised: A SYNC -RED-BG/SG and G M -RED.

Original: Each algorithm starts from the ﬁltered back-projection (FBP) of the measurements and runs for 1 hour.
Revised: Each algorithm starts from the ﬁltered back-projection (FBP) of the measurements and runs for 1 hour.

Original: Here, A SYNC -RED-SG randomly uses one-third of the total measurements at every iteration.
Revised: Here, A SYNC -RED-SG randomly uses one-third of the total measurements at every iteration.

Original: Given the same amount of time, A SYNC -RED-BG/SG successfully mitigates the noise-artifacts, while the result of G M -RED is still noisy.
Revised: Given the same amount of time, A SYNC -RED-BG/SG successfully mitigates the noise-artifacts, while the result of G M -RED is still noisy.

Original: In particular, the per-iteration time cost of A SYNC -RED-BG/SG and G M -RED is 5 .
Revised: In particular, the per-iteration time cost of A SYNC -RED-BG/SG and G M -RED is 5 .

Original: 23 , 3 . 21 , and 19 .
Revised: 23 , 3 . 21 , and 19 .

Original: 19 seconds, respectively.
Revised: 19 seconds, respectively.

Original: This experiment clearly illustrates the fast processing speed of the asynchronous procedure.
Revised: This experiment clearly illustrates the fast processing speed of the asynchronous procedure.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Asynchronous parallel methods have gained increasing importance in optimization for solving largescale imaging inverse problems.
Revised: Asynchronous parallel methods have gained increasing importance in optimization for solving largescale imaging inverse problems.

Original: We have introduced A SYNC -RED as an extension of the recent
Revised: We have introduced A SYNC -RED as an extension of the recent

Original: RED framework and theoretically analyze its convergence in batch and stochastic settings.
Revised: RED framework and theoretically analyze its convergence in batch and stochastic settings.

Original: We have validated its convergence guarantees and demonstrated its effectiveness in CT image reconstruction.
Revised: We have validated its convergence guarantees and demonstrated its effectiveness in CT image reconstruction.

Revised: We note that this work is complementary to traditional acceleration strategies, such as Nesterov acceleration and variance-reduction, commonly used in optimization.

Original: Future work will investigate  theoretical limits of A SYNC -RED in the unbounded maximal delay setting and explore its applicability to various inference problems in other data-intensive ﬁelds.
Revised: Future work will investigate A SYNC -RED with Nesterov acceleration (as was done in (Hannah et al., 2019) for traditional asynchronous block-coordinate algorithms) and variance-reduction (as was done in (Johnson & Zhang, 2013) for traditional stochastic gradient method) to better understand the tradeoffs between acceleration and scalability in multicore systems. We will additionally investigate theoretical limits of A SYNC -RED in the unbounded maximal delay setting.