Original: T RANSFERABLE R ECOGNITION -A WARE I MAGE P ROCESSING
Revised: T RANSFERABLE R ECOGNITION -A WARE I MAGE P ROCESSING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent progress in image recognition has stimulated the deployment of vision systems at an unprecedented scale.
Revised: Recent progress in image recognition has stimulated the deployment of vision systems at an unprecedented scale.

Original: As a result, visual data are now often consumed not only by humans but also by machines.
Revised: As a result, visual data are now often consumed not only by humans but also by machines.

Original: Existing image processing methods only optimize for better human perception, yet the resulting images may not be accurately recognized by machines.
Revised: Existing image processing methods only optimize for better human perception, yet the resulting images may not be accurately recognized by machines.

Original: This can be undesirable, e.g., the images can be improperly handled by search engines or recommendation systems.
Revised: This can be undesirable, e.g., the images can be improperly handled by search engines or recommendation systems.

Original: In this work, we propose simple approaches to improve machine interpretability of processed images: optimizing the recognition loss directly on the image processing network or through an intermediate transforming model.
Revised: In this work, we propose simple approaches to improve machine interpretability of processed images: optimizing the recognition loss directly on the image processing network or through an intermediate transforming model.

Original: Interestingly, the processing model’s ability to enhance recognition quality can transfer when evaluated on models of different architectures, recognized categories, tasks and training datasets.
Revised: Interestingly, the processing model’s ability to enhance recognition quality can transfer when evaluated on models of different architectures, recognized categories, tasks and training datasets.

Original: This makes the solutions applicable even when we do not have the knowledge of future recognition models, e.g., if we upload processed images to the Internet.
Revised: This makes the solutions applicable even when we do not have the knowledge of future recognition models, e.g., if we upload processed images to the Internet.

Original: We conduct experiments on multiple image processing tasks, with ImageNet classiﬁcation and PASCAL VOC detection as recognition tasks.
Revised: We conduct experiments on multiple image processing tasks, with ImageNet classiﬁcation and PASCAL VOC detection as recognition tasks.

Original: With our simple methods, substantial accuracy gain can be achieved with strong transferability and minimal image quality loss.
Revised: With our simple methods, substantial accuracy gain can be achieved with strong transferability and minimal image quality loss.

Original: Through a user study we further show that the accuracy gain can transfer to a black-box, third-party cloud model.
Revised: Through a user study we further show that the accuracy gain can transfer to a black-box, third-party cloud model.

Original: Finally, we try to explain this transferability phenomenon by demonstrating the similarities of different models’ decision boundaries.
Revised: Finally, we try to explain this transferability phenomenon by demonstrating the similarities of different models’ decision boundaries.

Original: Code is available at https://github.com/ anonymous20202020/Transferable_RA .
Revised: Code is available at https://github.com/ anonymous20202020/Transferable_RA .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unlike in image recognition where a neural network maps an image to a semantic label, a neural network used for image processing maps an input image to an output image with some desired properties.
Revised: Unlike in image recognition where a neural network maps an image to a semantic label, a neural network used for image processing maps an input image to an output image with some desired properties.

Original: Examples include super-resolution (Dong et al., 2014), denoising (Xie et al., 2012), deblurring (Eigen et al., 2013), colorization (Zhang et al., 2016).
Revised: Examples include super-resolution (Dong et al., 2014), denoising (Xie et al., 2012), deblurring (Eigen et al., 2013), colorization (Zhang et al., 2016).

Original: [Figure]
Revised: [Figure]

Original: The goal of such systems is to produce images of high perceptual quality to a human observer.
Revised: The goal of such systems is to produce images of high perceptual quality to a human observer.

Original: For example, in image denoising, we aim to remove noise that is not useful to an observer and restore the image to its original “clean” form.
Revised: For example, in image denoising, we aim to remove noise that is not useful to an observer and restore the image to its original “clean” form.

Original: Metrics like PSNR/SSIM (Wang et al., 2004) are often used (Dong et al., 2014; Tong et al., 2017) to approximate human-perceived similarity between the processed and original images, and direct human assessment on the ﬁdelity of the output is often considered the “gold-standard” (Ledig et al., 2017; Zhang et al., 2018b).
Revised: Metrics like PSNR/SSIM (Wang et al., 2004) are often used (Dong et al., 2014; Tong et al., 2017) to approximate human-perceived similarity between the processed and original images, and direct human assessment on the ﬁdelity of the output is often considered the “gold-standard” (Ledig et al., 2017; Zhang et al., 2018b).

Original: Therefore, techniques have been proposed to make outputs look perceptually pleasing to humans (Johnson et al., 2016; Ledig et al., 2017; Isola et al., 2017).
Revised: Therefore, techniques have been proposed to make outputs look perceptually pleasing to humans (Johnson et al., 2016; Ledig et al., 2017; Isola et al., 2017).

Original: However, while looking good to humans, image processing outputs may not be accurately recognized by image recognition systems.
Revised: However, while looking good to humans, image processing outputs may not be accurately recognized by image recognition systems.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the output image of an denoising model could easily be recognized by a human as a bird, but a recognition model classiﬁes it as a kite.
Revised: 1, the output image of an denoising model could easily be recognized by a human as a bird, but a recognition model classiﬁes it as a kite.

Original: One could  speciﬁcally train a recognition modelonly on these output images produced by the denoising model to achieve better performance on such images, or could leverage domain adaptation approaches to adapt the recognition model to this domain, but the performance on natural images can be harmed.
Revised: One could separate the image processing task and recognition task by speciﬁcally training a recognition model on these output images produced by the denoising model to achieve better performance on such images, or could leverage domain adaptation approaches to adapt the recognition model to this domain, but the performance on natural images can be harmed.

Original: This retraining/adaptation scheme might also be impractical considering the signiﬁcant overhead induced by catering to various image processing tasks and models.
Revised: This retraining/adaptation scheme might also be impractical considering the signiﬁcant overhead induced by catering to various image processing tasks and models.

Original: With the fast-growing size of image data, images are often “viewed” and analyzed more by machines than by humans.
Revised: With the fast-growing size of image data, images are often “viewed” and analyzed more by machines than by humans.

Original: Nowadays, any image uploaded to the Internet is likely to be analyzed by certain vision systems.
Revised: Nowadays, any image uploaded to the Internet is likely to be analyzed by certain vision systems.

Original: Therefore, it is of great importance for the processed images to be recognizable not only by humans, but also by machines.
Revised: Therefore, it is of great importance for the processed images to be recognizable not only by humans, but also by machines.

Original: In other words, recognition systems (e.g., image classiﬁer) should be able to accurately explain the underlying semantic meaning of the image content.
Revised: In other words, recognition systems (e.g., image classiﬁer) should be able to accurately explain the underlying semantic meaning of the image content.

Original: In this way, we make them easier to search, recommended to more interested audience, and so on, as these procedures are mostly executed by machines based on their understanding of the images.
Revised: In this way, we make them easier to search, recommended to more interested audience, and so on, as these procedures are mostly executed by machines based on their understanding of the images.

Original: Therefore, we argue that image processing systems should also aim at machine recognizability.
Revised: Therefore, we argue that image processing systems should also aim at machine recognizability.

Original: We call this problem “Recognition-Aware Image Processing”.
Revised: We call this problem “Recognition-Aware Image Processing”.

Original: It is also important that the enhanced recognizability is not speciﬁc to any concrete recognition model, i.e., the improvement is only achieved when the output images are evaluated on that particular model.
Revised: It is also important that the enhanced recognizability is not speciﬁc to any concrete recognition model, i.e., the improvement is only achieved when the output images are evaluated on that particular model.

Original: Instead, the improvement should ideally be transferable when evaluated on different models, to support its usage without access to possible future recognition systems, for example if we upload it to the Internet or share it on social media.
Revised: Instead, the improvement should ideally be transferable when evaluated on different models, to support its usage without access to possible future recognition systems, for example if we upload it to the Internet or share it on social media.

Revised: This is another case where we cannot separate the processing and recognition task by training them individually, since the recognition is not in our control.

Original: We may not know what network architectures (e.g. ResNet or VGG) will be used for inference, what object categories the model recognizes (e.g. animals or scenes), or even what task will be performed (e.g. classiﬁcation or detection).
Revised: We may not know what network architectures (e.g. ResNet or VGG) will be used for inference, what object categories the model recognizes (e.g. animals or scenes), or even what task will be performed (e.g. classiﬁcation or detection).

Original: Without these speciﬁcations, it might be hard to enhance image’s machine semantics.
Revised: Without these speciﬁcations, it might be hard to enhance image’s machine semantics.

Original: In this work, we introduce simple yet highly effective approaches to make image processing outputs more accurately recognized by downstream recognition systems, transferable among different recognition architectures, categories, tasks and training datasets.
Revised: In this work, we introduce simple yet highly effective approaches to make image processing outputs more accurately recognized by downstream recognition systems, transferable among different recognition architectures, categories, tasks and training datasets.

Original: The approaches we propose add a recognition loss optimized jointly with the image processing loss.
Revised: The approaches we propose add a recognition loss optimized jointly with the image processing loss.

Original: The recognition loss is computed using a ﬁxed recognition model that is pretrained on natural images, and can be done without further supervision from class labels for training images.
Revised: The recognition loss is computed using a ﬁxed recognition model that is pretrained on natural images, and can be done without further supervision from class labels for training images.

Original: It can be optimized either directly by the original image processing network, or through an intermediate transforming network.
Revised: It can be optimized either directly by the original image processing network, or through an intermediate transforming network.

Original: Interestingly, the accuracy gain transfers favorably among different recognition model architectures, object categories, and recognition tasks, which renders our simple solutions effective even when we do not have access to the recognition model.
Revised: Interestingly, the accuracy gain transfers favorably among different recognition model architectures, object categories, and recognition tasks, which renders our simple solutions effective even when we do not have access to the recognition model.

Original: Note that our contribution in these approaches does not lie in designing novel network architectures or training procedures, but in making the processed images more accurately recognized based on existing architectures/procedures.
Revised: Note that our contribution in these approaches does not lie in designing novel network architectures or training procedures, but in making the processed images more accurately recognized based on existing architectures/procedures.

Original: We also view our method’s simplicity mainly as a strength, as it is easy to be deployed and could serve as a baseline in this new problem.
Revised: We also view our method’s simplicity mainly as a strength, as it is easy to be deployed and could serve as a baseline in this new problem.

Original: We conduct extensive experiments, on multiple image processing (super-resolution, denoising, JPEGdeblocking) and downstream recognition (classiﬁcation, detection) tasks.
Revised: We conduct extensive experiments, on multiple image processing (super-resolution, denoising, JPEGdeblocking) and downstream recognition (classiﬁcation, detection) tasks.

Original: The results demonstrate our methods can substantially boost the recognition accuracy (e.g., up to 10%, or 20% relative gain), with minimal loss in image quality.
Revised: The results demonstrate our methods can substantially boost the recognition accuracy (e.g., up to 10%, or 20% relative gain), with minimal loss in image quality.

Original: Results are also compared with alternative approaches in Appendix 
Revised: Results are also compared with alternative approaches in Appendix A.

Original: A. We explore the transferability phenomenon in detail and demonstrate models’ decision boundary similarities to give an explanation. 
Revised: We explore the transferability phenomenon in different scenarios (architectures, categories, tasks/datasets, black-box models) and demonstrate models’ decision boundary similarities to give an explanation. To our best knowledge, this is the ﬁrst study on transferability of accuracy gain from the processing models trained with recognition loss.

Original: Our contributions can be summarized as:
Revised: Our contributions can be summarized as:

Original: • We propose to study the problem of enhancing the machine interpretability of image processing outputs, a desired property considering the amount of images analyzed by machines nowadays.
Revised: • We propose to study the problem of enhancing the machine interpretability of image processing outputs, a desired property considering the amount of images analyzed by machines nowadays.

Original: • We propose simple yet effective methods towards this goal, suitable for different use cases.
Revised: • We propose simple yet effective methods towards this goal, suitable for different use cases.

Original: Extensive experiments are conducted on multiple image processing and recognition tasks.
Revised: Extensive experiments are conducted on multiple image processing and recognition tasks.

Original: • We show that using our simple approaches, the recognition accuracy improvement could transfer among recognition architectures, categories, tasks and datasets, a desirable behavior making the proposed methods applicable without access to downstream recognition models.
Revised: • We show that using our simple approaches, the recognition accuracy improvement could transfer among recognition architectures, categories, tasks and datasets, a desirable behavior making the proposed methods applicable without access to downstream recognition models.

Original: • We provide decision boundary analysis of recognition models and show their similarities to gain a better understanding of the transferability phenomenon.
Revised: • We provide decision boundary analysis of recognition models and show their similarities to gain a better understanding of the transferability phenomenon.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Image processing/enhancement problems such as super-resolution and denoising have a long history (Tsai, 1984;
Revised: Image processing/enhancement problems such as super-resolution and denoising have a long history (Tsai, 1984;

Original: Park et al., 2003; Rudin et al., 1992; Candès et al., 2006).
Revised: Park et al., 2003; Rudin et al., 1992; Candès et al., 2006).

Original: Since the initial success of deep neural networks on these problems (Dong et al., 2014; Xie et al., 2012; Wang et al., 2016b), a large body of works try to investigate better model architecture design and training techniques 
Revised: Since the initial success of deep neural networks on these problems (Dong et al., 2014; Xie et al., 2012; Wang et al., 2016b), a large body of works try to investigate better model architecture design and training techniques (Dong et al., 2016;

Original: (Dong et al., 2016; Kim et al., 2016b; Shi et al., 2016;
Revised: Kim et al., 2016b; Shi et al., 2016;

Original: Kim et al., 2016a; Lai et al., 2017; Chen et al., 2018)
Revised: Kim et al., 2016a; Lai et al., 2017; Chen et al., 2018)

Original: These works focus on generating high visual quality images under conventional metrics (PSNR/SSIM) or human evaluation, without considering recognition performance on the output.
Revised: These works focus on generating high visual quality images under conventional metrics (PSNR/SSIM) or human evaluation, without considering recognition performance on the output.

Original: There are also a number of works that relate image recognition with image processing.
Revised: There are also a number of works that relate image recognition with image processing.

Original: Some works (Zhang et al., 2016; Larsson et al., 2016; Zhang et al., 2018c; Sajjadi et al., 2017; Lee et al., 2019) use image recognition accuracy as an evaluation metric for image colorization/super-resolution/denoising, but without optimizing for it in training.
Revised: Some works (Zhang et al., 2016; Larsson et al., 2016; Zhang et al., 2018c; Sajjadi et al., 2017; Lee et al., 2019) use image recognition accuracy as an evaluation metric for image colorization/super-resolution/denoising, but without optimizing for it in training.

Original: Wang et al.
Revised: Wang et al.

Original: (2016a)
Revised: (2016a)

Original: ; Haris et al.

Original: (2018a)

Original: ; VidalMata et al.
Revised: ; VidalMata et al.

Original: ; Banerjee et al.
Revised: ; Banerjee et al.

Original: (2019) investigate how to achieve more accurate recognition on lowresolution or corrupted/noisy images.
Revised: (2019) investigate how to achieve more accurate recognition on low-resolution or corrupted/noisy images, but did not resort to the use of recognition loss.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) propose a method to make denoised images more accurately segmented.
Revised: (2019) propose a method to make denoised images more accurately segmented.

Original: Liu et al.
Revised: Liu et al.

Original: (2019) introduced a theoretical framework for classiﬁcationdistortion-perception tradeoff and conducted experiments with simulated or toy datasets, while our work develops practical approaches for real-world datasets.
Revised: (2019) introduced a theoretical framework for classiﬁcation-distortion-perception tradeoff and conducted experiments with simulated or toy datasets, while our work develops practical approaches for real-world datasets.

Original: Most existing works only consider one image processing task or image domain, and develop speciﬁc techniques, while our simpler approach is task-agnostic, more widely applicable, and is ﬁrst to be shown transferable.
Revised: Most existing works only consider one image processing task or image domain, and develop speciﬁc techniques, while our simpler approach is task-agnostic and more widely applicable.

Original: Our work is related but different from those aiming for robustness of the recognition model (Hendrycks & Dietterich, 2019; Li et al., 2019; Shankar et al., 2018), as we focus on training the processing models.
Revised: Our work is related but different from those aiming for robustness of the recognition model (Hendrycks & Dietterich, 2019; Li et al., 2019; Shankar et al., 2018), as we focus on training the processing models.

Original: Our method also shares some similarity with Palacio et al.
Revised: Our method also shares some similarity with Palacio et al.

Original: (2018) which tries to differentiate input signals by optimizing recognition accuracy using auto-encoders.
Revised: (2018) which tries to differentiate input signals by optimizing recognition accuracy using auto-encoders.

Original: Bai et al.
Revised: Bai et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018a)
Revised: (2018a);

Original: ; Liu et al.
Revised: Liu et al.

Original: ; Sharma et al.
Revised: ; Sharma et al.

Original: ; Bai et al.
Revised: ; Bai et al.

Original: Liu et al.
Revised: Liu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2018) jointly train a processing model (e.g., dehazing, face reconstruction) together with a recognition model (e.g., face recognition) to achieve better image processing and/or recognition quality.
Revised: (2018) jointly train a processing model (e.g., dehazing, face reconstruction) together with a recognition model (e.g., face recognition) to achieve better image processing and/or recognition quality.

Original: Our problem setting is different from these works, in that we assume we do not have the control of the recognition model, as it might be on the cloud or to be decided in the future, thus we adapt the image processing model only.
Revised: Our problem setting is different from these works, in that we assume we do not have the control of the recognition model, as it might be on the cloud or to be decided in the future, thus we adapt the image processing model only.

Original: This provides transferability and also ensures the recognition of natural images is not harmed.
Revised: This provides transferability and also ensures the recognition of natural images is not harmed.

Revised: To our knowledge, our work is the ﬁrst to analyze such transferability in the literature.

Original: Appendix A includes our comparison with training recognition model jointly.
Revised: Appendix A includes our comparison with training recognition model jointly.

Revised: We design and compare variants of our methods for different use cases, while prior studies only consider a single case.

Revised: In addition, using extra discriminators have also been used in few-shot learning (Gidaris et al., 2019; Zhang et al., 2020) to enhance the feature quality and improve the performance on the original few-shot task, while our use of recognition model as loss is to improve image processing outputs’ recognition accuracy, instead of the original processing task.

Original:  M ETHOD
Revised:  M ETHOD

Original: We ﬁrst introduce the problem setting of “recognition-aware” image processing, and then develop various approaches, each suited for different use cases.
Revised: We ﬁrst introduce the problem setting of “recognition-aware” image processing, and then develop various approaches, each suited for different use cases.

Original: Our proposed methodology, although only introduced in a vision context, can be extended to other domains (e.g., speech) as well.
Revised: Our proposed methodology, although only introduced in a vision context, can be extended to other domains (e.g., speech) as well.

Original: Problem Setting.
Revised: Problem Setting.

Original: In a typical image processing problem, given a set of training input images { I kin } and corresponding target images { I ktarget } , we aim to train a network that maps an input to its target.
Revised: In a typical image processing problem, given a set of training input images { I kin } and corresponding target images { I ktarget } , we aim to train a network that maps an input to its target.

Original: Denoting this network as P (processing), parameterized by W P , our optimization objective is:
Revised: Denoting this network as P (processing), parameterized by W P , our optimization objective is:

Original: [Equation]
Revised: [Equation]

Original: where l proc is the loss function for each sample (e.g., L 2 ).
Revised: where l proc is the loss function for each sample (e.g., L 2 ).

Original: The performance is typically evaluated by similarity (e.g., PSNR/SSIM) between I ktarget and P (cid:0)
Revised: The performance is typically evaluated by similarity (e.g., PSNR/SSIM) between I k target and P (cid:0)

Original: I kin (cid:1) , or human assessment.
Revised: I k in (cid:1) , or human assessment.

Original: In recognitionaware processing, we are interested in a recognition task, with a trained recognition model R (recognition).
Revised: In recognitionaware processing, we are interested in a recognition task, with a trained recognition model R (recognition).

Original: We assume each target image I ktarget is associated with a semantic label S k for the recognition task.
Revised: We assume each target image I ktarget is associated with a semantic label S k for the recognition task.

Original: Our goal is to train a processing model P such that the recognition performance on the output images P (cid:0) I kin (cid:1) is high, when evaluated using R with the semantic labels { S k } .
Revised: Our goal is to train a processing model P such that the recognition performance on the output images P (cid:0) I kin (cid:1) is high, when evaluated using R with the semantic labels { S k } .

Original: In practice, R might not be available (e.g., on the cloud), in which case we could resort to other models if the performance improvement transfers among models.
Revised: In practice, R might not be available (e.g., on the cloud), in which case we could resort to other models if the performance improvement transfers among models.

Original: Optimizing Recognition Loss.
Revised: Optimizing Recognition Loss.

Original: Given our goal is to make the output images by P more recognizable  by R , it is natural to add a recognition loss on top of the objective of the image processing task (Eqn.
Revised: Given our goal is to make the output images by P more recognizable [Figure] by R , it is natural to add a recognition loss on top of the objective of the image processing task (Eqn.

Original: 1) during training:
Revised: 1) during training:

Original: [Equation]
Revised: [Figure]

Original:  l recog is the per-example recognition loss deﬁned by the downstream recognition task.
Revised: (2) l recog is the per-example recognition loss deﬁned by the downstream recognition task.

Original: For example, for image classiﬁcation, l recog could be the cross-entropy (CE) loss.
Revised: For example, for image classiﬁcation, l recog could be the cross-entropy (CE) loss.

Original: Adding the image processing loss (Eqn. 1) and recognition loss (Eqn. 2) together, our total training objective becomes
Revised: Adding the image processing loss (Eqn. 1) and recognition loss (Eqn. 2) together, our total training objective becomes

Original: [Equation]
Revised: [Figure]

Original:  where λ is the coefﬁcient controlling the weights of L recog relative to L proc .
Revised: (3) where λ is the coefﬁcient controlling the weights of L recog relative to L proc .

Original: We denote this simple solution as “RA (Recognition-Aware) processing”, which is visualized in Fig.
Revised: We denote this simple solution as “RA (Recognition-Aware) processing”, which is visualized in Fig.

Original: 2 left.
Revised: 2 left.

Original: Note that[Figure] once the training is ﬁnished, the recognition model used as loss is not needed anymore, and during inference, we only need the processing model P, thus no overhead is introduced in deployment.
Revised: Note that once the training is ﬁnished, the recognition model used as loss is not needed anymore, and during inference, we only need the processing model P, thus no overhead is introduced in deployment.

Original: A potential shortcoming of directly optimizing L recog is that it might deviate P from optimizing the original loss L proc , and the trained P will generate images that are not as good as if we only optimize L proc .
Revised: A potential shortcoming of directly optimizing L recog is that it might deviate P from optimizing the original loss L proc , and the trained P will generate images that are not as good as if we only optimize L proc .

Original: We will show that, however, with proper choice of λ , we could substantially boost the recognition performance with nearly no sacriﬁce on image quality.
Revised: We will show that, however, with proper choice of λ , we could substantially boost the recognition performance with nearly no sacriﬁce on image quality.

Original: If using R as a ﬁxed loss can only boost the accuracy on R itself, the use of the method could be limited.
Revised: If using R as a ﬁxed loss can only boost the accuracy on R itself, the use of the method could be limited.

Original: Sometimes we do not have the knowledge about the future downstream recognition model or even task.
Revised: Sometimes we do not have the knowledge about the future downstream recognition model or even task.

Original: Interestingly, we ﬁnd that processing models trained with the loss of one recognition model
Revised: Interestingly, we ﬁnd that processing models trained with the loss of one recognition model

Original: R 1 , can also boost the performance when evaluated using another model R 2 , even if model R 2 has a different architecture, recognizes different categories or even performs a different task.
Revised: R 1 , can also boost the performance when evaluated using another model R 2 , even if model R 2 has a different architecture, recognizes different categories or even performs a different task.

Original: This makes our method effective even when we cannot access the target downstream model, where we could use another trained model as the loss function.
Revised: This makes our method effective even when we cannot access the target downstream model, where we could use another trained model as the loss function.

Original: This phenomenon also implies that the “recognizability” of a processed image can be more general than just the extent it ﬁts to a speciﬁc model.
Revised: This phenomenon also implies that the “recognizability” of a processed image can be more general than just the extent it ﬁts to a speciﬁc model.

Original: Unsupervised Optimization of Recognition Loss.
Revised: Unsupervised Optimization of Recognition Loss.

Original: The solution above requires semantic labels for training images, which however, may not be always available.
Revised: The solution above requires semantic labels for training images, which however, may not be always available.

Original: In this case, we could regress the recognition model’s output of the target image R ( I ktarget ) , The recognition objective changes to
Revised: In this case, we could regress the recognition model’s output of the target image R ( I ktarget ) , The recognition objective changes to

Original: [Equation]
Revised: [Equation]

Original: where l dis is a distance measure between two of R ’s outputs (e.g., L 2 distance, KL divergence).
Revised: where l dis is a distance measure between two of R ’s outputs (e.g., L 2 distance, KL divergence).

Original: We call this approach “unsupervised RA”.
Revised: We call this approach “unsupervised RA”.

Original: Note that it is only unsupervised for training model P , but not necessarily for the model R .
Revised: Note that it is only unsupervised for training model P , but not necessarily for the model R .

Original: The (pre)training of the model R is not our concern since in our problem setting R is a given trained model, and it can be trained either with or without full supervision.
Revised: The (pre)training of the model R is not our concern since in our problem setting R is a given trained model, and it can be trained either with or without full supervision.

Original: Unsupervised RA is related to the “knowledge distillation” paradigm (Hinton et al., 2014) used for network model compression, where the output of a large model is used to guide a small model, given the same input images.
Revised: Unsupervised RA is related to the “knowledge distillation” paradigm (Hinton et al., 2014) used for network model compression, where the output of a large model is used to guide a small model, given the same input images.

Original: Instead we use the same recognition model R , but guide the upstream processing model to generate desirable images.
Revised: Instead we use the same recognition model R , but guide the upstream processing model to generate desirable images.

Original: It is also related to the perceptual loss/feature loss used in Johnson et al.
Revised: It is also related to the perceptual loss/feature loss used in Johnson et al.

Original: ; Ledig et al.
Revised: ; Ledig et al.

Original: (2017), where the feature distance is minimized instead of output distance.
Revised: (2017), where the feature distance is minimized instead of output distance.

Original: We provide a comparison in Appendix A.
Revised: We provide a comparison in Appendix A.

Original: Using an Intermediate Transformer.
Revised: Using an Intermediate Transformer.

Original: Sometimes we want to prevent the added recognition loss
Revised: Sometimes we want to prevent the added recognition loss

Original: L recog from causing P to deviate from optimizing its original loss.
Revised: L recog from causing P to deviate from optimizing its original loss.

Original: We can achieve this by introducing an intermediate transformation model T : P ’s output is ﬁrst fed to the T , and T ’s output serves as the input for R (Fig. 2 right).
Revised: We can achieve this by introducing an intermediate transformation model T : P ’s output is ﬁrst fed to the T , and T ’s output serves as the input for R (Fig. 2 right).

Original: T ’s parameters W T are optimized for the recognition loss:
Revised: T ’s parameters W T are optimized for the recognition loss:

Original: [Equation]
Revised: [Equation]

Original: With the help of T on optimizing the recognition loss, the model P can now “focus” on its original image processing loss L proc .
Revised: With the help of T on optimizing the recognition loss, the model P can now “focus” on its original image processing loss L proc .

Original: The optimization objective becomes:
Revised: The optimization objective becomes:

Original: [Equation]
Revised: [Equation]

Original: In Eqn. 6, P is solely optimizing L proc as in the original image processing problem (Eqn. 1).
Revised: In Eqn. 6, P is solely optimizing L proc as in the original image processing problem (Eqn. 1).

Original: P is learned as if there is no recognition loss, and therefore the image processing quality of its output will not be affected.
Revised: P is learned as if there is no recognition loss, and therefore the image processing quality of its output will not be affected.

Original: This could be implemented by “detaching” the gradient generated by L recog between the model T and P (Fig. 2 right).
Revised: This could be implemented by “detaching” the gradient generated by L recog between the model T and P (Fig. 2 right).

Original: We term this solution as “RA with transformer”.
Revised: We term this solution as “RA with transformer”.

Original: Its downside compared with directly optimization using P is that there are two instances for each image (the output of model P and T ), one is “for human” and the other is “for machines”.
Revised: Its downside compared with directly optimization using P is that there are two instances for each image (the output of model P and T ), one is “for human” and the other is “for machines”.

Original: Therefore, the transformer is best suited when we want to guarantee the image processing quality not affected at all, at the expense of maintaining another image.
Revised: Therefore, the transformer is best suited when we want to guarantee the image processing quality not affected at all, at the expense of maintaining another image.

Original: For example, in classifying images, we can have the higher-quality image presented to users for better experience and the other image passed to the backend for accurate machine classiﬁcation.
Revised: For example, in classifying images, we can have the higher-quality image presented to users for better experience and the other image passed to the backend for accurate machine classiﬁcation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We experiment on three image processing tasks: image super-resolution (SR), denoising, and JPEGdeblocking (17 tasks from ImageNet-C (Hendrycks & Dietterich, 2019) in Appendix F).
Revised: We experiment on three image processing tasks: image super-resolution (SR), denoising, and JPEGdeblocking (17 tasks from ImageNet-C (Hendrycks & Dietterich, 2019) in Appendix F).

Original: To obtain the input images, for SR, we downsample images by a factor of 4 × ; for denoising, we add Gaussian noise with 0.1 standard deviation; for JPEG deblocking, a quality factor of 10 is used for compression.
Revised: To obtain the input images, for SR, we downsample images by a factor of 4 × ; for denoising, we add Gaussian noise with 0.1 standard deviation; for JPEG deblocking, a quality factor of 10 is used for compression.

Original: We pair these three tasks with two visual recognition tasks, ImageNet classiﬁcation andPASCAL
Revised: We pair these three tasks with two visual recognition tasks, ImageNet classiﬁcation and

Original:  VOC object detection.
Revised: PASCAL VOC object detection.

Original: We adopt SRResNet
Revised: We adopt SRResNet

Original: (Ledig et al., 2017) as the architecture of the image processing model P (more architectures in Appendix C).
Revised: (Ledig et al., 2017) as the architecture of the image processing model P (more architectures in Appendix C).

Original: For the transformer model T , we use the ResNet-like architecture in CycleGAN (Zhu et al., 2017).
Revised: For the transformer model T , we use the ResNet-like architecture in CycleGAN (Zhu et al., 2017).

Original: The recognition models are ResNet, VGG and DenseNet.
Revised: The recognition models are ResNet, VGG and DenseNet.

Original: Please refer to Appendix A for comparison with alternative approaches and B for more details on training/hyperparameter settings.
Revised: Please refer to Appendix A for comparison with alternative approaches and B for more details on training/hyperparameter settings.

Original: Evaluation on the Same Recognition Model.
Revised: Evaluation on the Same Recognition Model.

Original: We ﬁrst present results when the R used for evaluation is the same as the R we use as the recognition loss in training.
Revised: We ﬁrst present results when the R used for evaluation is the same as the R we use as the recognition loss in training.

Original: Table 1 shows our results on
Revised: Table 1 shows our results on

Original: ImageNet classiﬁcation.
Revised: ImageNet classiﬁcation.

Original: ImageNet-pretrained classiﬁcation models ResNet-18/50/101, DenseNetand VGG-16 are denoted as R18/50/101, D121, V16.
Revised: ImageNet-pretrained classiﬁcation models ResNet-18/50/101, DenseNetand VGG-16 are denoted as R18/50/101, D121, V16.

Original: “No Processing” denotes the accuracy on input images (low-resolution/noisy/JPEG-compressed); “Plain Processing” denotes using image processing models trained without recognition loss (Eqn. 1).
Revised: “No Processing” denotes the accuracy on input images (low-resolution/noisy/JPEG-compressed); “Plain Processing” denotes using image processing models trained without recognition loss (Eqn. 1).

Original: We observe that plain processing can boost the accuracy over unprocessed images.
Revised: We observe that plain processing can boost the accuracy over unprocessed images.

Original: These two are considered as baselines.
Revised: These two are considered as baselines.

Original: [Table] From Table
Revised: Note there exist instances where the unprocessed images are classiﬁed correctly but the plain processing outputs are classiﬁed incorrectly.

Original: [Table]
Revised: For example, on super-resolution with ResNet-18 as the recognition model, these instances account for 6.14% of the validation set.

Original: using RA processing can signiﬁcantly boost the  accuracy of output images over plainly processed ones
Revised: However, plain image processing does help the recognition accuracy on average.

Original: [Table]
Revised: [Table]

Original:  for all image processing tasks and recognition models.
Revised: From Table 1, using RA processing can signiﬁcantly boost the accuracy of output images over plainly processed ones, for all image processing tasks and recognition models.

Original: This is more prominent when the accuracy of plain processing is lower, e.g., in SR and JPEG-deblocking, where we mostly obtain ∼ 10% accuracy gain (close to 20% in relative terms).
Revised: This is more prominent when the accuracy of plain processing is lower, e.g., in SR and JPEG-deblocking, where we mostly obtain ∼ 10% accuracy gain (close to 20% in relative terms).

Original: Even without semantic labels, our unsupervised
Revised: Even without semantic labels, our unsupervised

Original: RA can still in most cases outperform baseline methods, despite achieves lower accuracy than its supervised counterpart.
Revised: RA can still in most cases outperform baseline methods, despite achieves lower accuracy than its supervised counterpart.

Original: Also in SR and JPEG-deblocking, using an intermediate transformer T can bring additional improvement over RA processing.
Revised: Also in SR and JPEG-deblocking, using an intermediate transformer T can bring additional improvement over RA processing.

Original: See Appendix D for results on detection.
Revised: See Appendix D for results on detection.

Original: Transfer between Recognition Architectures.
Revised: Transfer between Recognition Architectures.

Original: In reality, sometimes the R we want to eventually evaluate the output images on might not be available for us to use as a loss for training, e.g., it could be on the cloud, kept conﬁdential or decided later.
Revised: In reality, sometimes the R we want to eventually evaluate the output images on might not be available for us to use as a loss for training, e.g., it could be on the cloud, kept conﬁdential or decided later.

Original: In this case, we could train an processing model P using recognition model R A (source) that is accessible to us, and after we obtain the trained model P , evaluate its output images’ accuracy using another unseen R B (target).
Revised: In this case, we could train an processing model P using recognition model R A (source) that is accessible to us, and after we obtain the trained model P , evaluate its output images’ accuracy using another unseen R B (target).

Original: We evaluate model architecture pairs on ImageNet in Table 2, for RA Processing, where row is the source model ( R A ), and column is the target model ( R B ).
Revised: We evaluate model architecture pairs on ImageNet in Table 2, for RA Processing, where row is the source model ( R A ), and column is the target model ( R B ).

Original: In Table 2’s each column, training with any model R A produces substantially higher accuracy than plainly processed images on R B , indicating that the improvement is transferable among recognition architectures.
Revised: In Table 2’s each column, training with any model R A produces substantially higher accuracy than plainly processed images on R B , indicating that the improvement is transferable among recognition architectures.

Original: This phenomenon enables us to use RA processing without the knowledge of the downstream recognition architecture.
Revised: This phenomenon enables us to use RA processing without the knowledge of the downstream recognition architecture.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Transfer between Object Categories.
Revised: Transfer between Object Categories.

Original: What if the R A and R B recognize different categories of objects?
Revised: What if the R A and R B recognize different categories of objects?

Original: We divide the 1000 classes from
Revised: We divide the 1000 classes from

Original: ImageNet into two splits, denoted as Cat (category)
Revised: ImageNet into two splits, denoted as Cat (category)

Original: A/B , each with 500 classes, and train two 500-way classiﬁers (R18) on both splits, obtaining R A and R B .
Revised: A/B , each with 500 classes [Table] and train two 500-way classiﬁers (R18) on both splits, obtaining R A and R B .

Original: Next, we train two image processing models P A /P
Revised: Next, we train two image processing models P A /P

Original: B with the R A /R B as recognition loss, using images from Cat A/B respectively.
Revised: B with the R A /R B as recognition loss, using images from Cat A/B respectively.

Original: Note that neither P nor R has seen images/categories from the other split.
Revised: Note that neither P nor R has seen images/categories from the other split.

Original: We evaluate obtained processing models on both splits in Table 3.
Revised: We evaluate obtained processing models on both splits in Table 3.

Original: We observe that RA still beneﬁts the accuracy even when transferring across categories (e.g., in SR, 60.1% to 66.5% transferring from Cat A to Cat B ).
Revised: We observe that RA still beneﬁts the accuracy even when transferring across categories (e.g., in SR, 60.1% to 66.5% transferring from Cat A to Cat B ).

Original: The improvement is only marginally lower than directly training on the same categories (e
Revised: The improvement is only marginally lower than directly training on the same categories (e.g., 60.2% to 67.8% on Cat B ). This suggests

Revised: RA processing models do not impose category-speciﬁc signals to the images

Original: [Table]
Revised: [Table]

Original: g., 60.2% to 67.8% on Cat B ). This suggests RA processing models do not impose category-speciﬁc signals to the images, but signals that enable wider sets of classes to be better recognized. 
Revised: but signals that enable wider sets of classes to be better recognized. [Table] w/ V

Original: [Table]
Revised: [Table]

Original: Transfer between Recognition Tasks and Datasets.
Revised: Transfer between Recognition Tasks and Datasets.

Original: We evaluate task transferability when task A is classiﬁcation and task B is object detection in Table 4, where rows are classiﬁcation models used for RA loss and columns are detection models for evaluation.
Revised: We evaluate task transferability when task A is classiﬁcation and task B is object detection in Table 4, where rows are classiﬁcation models used for RA loss and columns are detection models for evaluation.

Original: There is also a dataset shift, since model P and R are both trained on ImageNet; during evaluation, P is fed with VOC images and we use a VOC-trained detection model R .
Revised: There is also a dataset shift, since model P and R are both trained on ImageNet; during evaluation, P is fed with VOC images and we use a VOC-trained detection model R .

Original: We observe that using classiﬁcation loss on model A (row) gives accuracy gain on model B over plain processing in most cases.
Revised: We observe that using classiﬁcation loss on model A (row) gives accuracy gain on model B over plain processing in most cases.

Original: Such task transferability suggests the “machine semantics” of the image could be a task-agnostic property.
Revised: Such task transferability suggests the “machine semantics” of the image could be a task-agnostic property.

Original: Transfer to an Black-box, Third-party Cloud Model.
Revised: Transfer to an Black-box, Third-party Cloud Model.

Original: We compare the images generated from plain processing and RA models using the “General” model at clarifai.com , a company providing state-of-the-art image classiﬁcation cloud services.
Revised: We compare the images generated from plain processing and RA models using the “General” model at clarifai.com , a company providing state-of-the-art image classiﬁcation cloud services.

Original: We do not have knowledge of the model’s architecture or what datasets it was trained on, except we can access the service using APIs.
Revised: We do not have knowledge of the model’s architecture or what datasets it was trained on, except we can access the service using APIs.

Original: The model also recognizes over 11000+ concepts that are different from the 1000-class ImageNet categories.
Revised: The model also recognizes over 11000+ concepts that are different from the 1000-class ImageNet categories.

Original: For this experiment, we only take the output category with the maximum probability as the prediction.
Revised: For this experiment, we only take the output category with the maximum probability as the prediction.

Original: We use the SR processing model trained with R18/ImageNet as the RA model.
Revised: We use the SR processing model trained with R18/ImageNet as the RA model.

Original: We randomly sample image indices from ImageNet validation set, and ask clarifai.com for predictions of both images generated from plain and RA processing models.
Revised: We randomly sample image indices from ImageNet validation set, and ask clarifai.com for predictions of both images generated from plain and RA processing models.

Original: From the results, we then randomly select 100 instances where clarifai.com gives different predictions on plain and RA images, to compose a survey for user study.
Revised: From the results, we then randomly select 100 instances where clarifai.com gives different predictions on plain and RA images, to compose a survey for user study.

Original: For each of the 100 instances, the survey presents the user with the target image, and both prediction labels generated from plain/RA images, in randomized left/right order.
Revised: For each of the 100 instances, the survey presents the user with the target image, and both prediction labels generated from plain/RA images, in randomized left/right order.

Original: The survey asks the user to indicate in his/her opinion which label(s) describe the image to a satisfactory level.
Revised: The survey asks the user to indicate in his/her opinion which label(s) describe the image to a satisfactory level.

Original: The user has the options to choose none, either or both labels.
Revised: The user has the options to choose none, either or both labels.

Original: The survey and instructions can be found at https://tinyurl.com/y698779q .
Revised: The survey and instructions can be found at https://tinyurl.com/y698779q .

Original: 10 volunteers participated in our survey.
Revised: 10 volunteers participated in our survey.

Original: The resulting average satisfaction rates for plain and RA super-resolved images are 40.1% and 55.3% respectively.
Revised: The resulting average satisfaction rates for plain and RA super-resolved images are 40.1% and 55.3% respectively.

Original: We achieve 15.2% absolute gain or 37.9% relative gain on recognition satisﬁcation rate, indicating the strong transferability our method provides without knowledge of the black-box cloud model.
Revised: We achieve 15.2% absolute gain or 37.9% relative gain on recognition satisﬁcation rate, indicating the strong transferability our method provides without knowledge of the black-box cloud model.

Original:  A NALYSIS AND C ONCLUSION
Revised:  A NALYSIS AND C ONCLUSION

Original: [Table]
Revised: [Table]

Original: Image Processing Quality Assessment.
Revised: Image Processing Quality Assessment.

Original: We compare the output image quality using conventional metrics (PSNR/SSIM).
Revised: We compare the output image quality using conventional metrics (PSNR/SSIM).

Original: When using RA with transformer, the output quality of P is guaranteed unaffected, therefore here we evaluate RA processing.
Revised: When using RA with transformer, the output quality of P is guaranteed unaffected, therefore here we evaluate RA processing.

Original: We use R18 as loss on ImageNet, and report results with different λ s (Eqn. 3) in Table 5.
Revised: We use R18 as loss on ImageNet, and report results with different λ s (Eqn. 3) in Table 5.

Original: λ = 0 corresponds to plain processing.
Revised: λ = 0 corresponds to plain processing.

Original: When λ = 10 − 4 , PSNR/SSIM are only marginally worse.
Revised: When λ = 10 − 4 , PSNR/SSIM are only marginally worse.

Original: However, the accuracy obtained is signiﬁcantly higher.
Revised: However, the accuracy obtained is signiﬁcantly higher.

Original: This suggests that the added recognition loss is not harmful when λ is chosen properly.
Revised: This suggests that the added recognition loss is not harmful when λ is chosen properly.

Original: When λ is excessively large ( 10 − 2 ), image quality is hurt more, and interestingly even the recognition accuracy start to decrease, which could be due to the change of actual learning rate.
Revised: When λ is excessively large ( 10 − 2 ), image quality is hurt more, and interestingly even the recognition accuracy start to decrease, which could be due to the change of actual learning rate.

Original: A proper balance between processing and recognition loss is needed for both image quality and accuracy.
Revised: A proper balance between processing and recognition loss is needed for both image quality and accuracy.

Original: We also measure the image quality using the PieAPP metric (Prashnani et al., 2018), which emphasizes more on perceptual difference: on SR, when λ = 0/10 − 4 /10 − 3 , PieAPP (lower is better)
Revised: We also measure the image quality using the PieAPP metric (Prashnani et al., 2018), which emphasizes more on perceptual difference: on SR

Original: = 1.329/1.313/1.323.

Original: Interestingly, RA processing can slightly improve perceptual quality measured with PieAPP.[Figure]
Revised: Interestingly, RA processing can slightly improve perceptual quality measured with PieAPP.

Original: In Fig.
Revised: In Fig.

Original: 3, we visualize some examples where the output image is incorrectly classiﬁed with a plain processing model, but correctly recognized with RA processing.
Revised: 3, we visualize some examples where the output image is incorrectly classiﬁed with a plain processing model, but correctly recognized with RA processing.

Original: With smaller λ ( 10 − 2 and 10 − 3 ), the image is nearly the same as the plainly processed images.
Revised: With smaller λ ( 10 − 2 and 10 − 3 ), the image is nearly the same as the plainly processed images.

Original: When λ is too large ( 10 − 2 ), we could see some extra textures when zooming in.
Revised: When λ is too large ( 10 − 2 ), we could see some extra textures when zooming in.

Original: More results are presented in Appendix G.
Revised: More results are presented in Appendix H.

Original: Decision Boundaries Analysis and Transferability.
Revised: Decision Boundaries Analysis and Transferability.

Original: Inspired by prior works’ analysis on adversarial example transferability (Liu et al., 2016; Tramèr et al., 2017), we conduct decision boundary analysis to gain insights on RA processing’s transferability.
Revised: Inspired by prior works’ analysis on adversarial example transferability (Liu et al., 2016; Tramèr et al., 2017), we conduct decision boundary analysis to gain insights on RA processing’s transferability.

Original: The task used is SR with ImageNet.
Revised: The task used is SR with ImageNet.

Original: We restrict our analysis to a single direction at a time due to image’s high dimension: given a input image x and a direction d (unit vector, same dimension as x ), we analyze how the output of the recognition model R changes when
Revised: We restrict our analysis to a single direction at a time due to image’s high dimension: given a input image x and a direction d (unit vector, same dimension as x ), we analyze how the output of the recognition model R changes when

Original: x moves along d by δ amount, i.e., when input is x + δs .
Revised: x moves along d by δ amount, i.e., when input is x + δs .

Original:  We deﬁne the boundary distance (BD) of model R , with respect to input x and direction s , as the minimum amount of movement along d required for x to produce a different label at R , or more formally: BD ( R, x, d ) =
Revised: [Figure] We deﬁne the boundary distance (BD) of model R , with respect to input x and direction s , as the minimum amount of movement along d required for x to produce a different label at R , or more formally: BD ( R, x, d ) =

Original: arg min δ> 0 { R ( x + δ · d ) (cid:54) =
Revised: arg min δ> 0 { R ( x + δ · d ) (cid:54) =

Original: R ( x ) } .
Revised: R ( x ) } .

Original: Consider a two-model scenario, with a source model R s and a target model R t sharing the same output categories.
Revised: Consider a two-model scenario, with a source model R s and a target model R t sharing the same output categories.

Original: We deﬁne their inter-boundary distance (IBD): IBD ( R s , R t , x, d ) =
Revised: We deﬁne their inter-boundary distance (IBD): IBD ( R s , R t , x, d ) =

Original: | BD ( R s , x, d ) −
Revised: | BD ( R s , x, d ) −

Original: BD ( R t , x, d ) | .
Revised: BD ( R t , x, d ) | .

Original: Intuitively, if R s ( x ) =
Revised: Intuitively, if R s ( x ) =

Original: R t ( x ) (same prediction within boundary), a small IBD between R s and R t means they have close boundaries along the s direction, since x does not need to move beyond one’s boundary too far to reach the other’s.
Revised: R t ( x ) (same prediction within boundary), a small IBD between R s and R t means they have close boundaries along the s direction, since x does not need to move beyond one’s boundary too far to reach the other’s.

Original: In this case, changes made to x along d likely has a transferring effect from source to target model due to their close boundaries. 
Revised: In this case, changes made to x along d likely has a transferring effect from source to target model due to their close boundaries. Direction

Original: [Table]
Revised: [Table]

Revised: IBD ( R s , R t ) 25.

Original: We take the image x to be a plain processing output, and consider two types of directions: 1.
Revised: We take the image x to be a plain processing output, and consider two types of directions: 1.

Original: random direction d r . 2.
Revised: random direction d r . 2.

Original: The direction generated by subtracting the plain processing output x from RA processing output x s , i.e., ( x s − x ) /
Revised: The direction generated by subtracting the plain processing output x from RA processing output x s , i.e., ( x s − x ) /

Original: || x s − x || 2 .
Revised: || x s − x || 2 .

Original: The RA processing model here is trained with the source model R s , thus x s is speciﬁc to R s .
Revised: The RA processing model here is trained with the source model R s , thus x s is speciﬁc to R s .

Original: We call this “RA direction” ( d RA ), since it points to the RA output x s from the plain output x .
Revised: We call this “RA direction” ( d RA ), since it points to the RA output x s from the plain output x .

Original: We take all validation images such that the plain processing output x generates the same wrong prediction when fed to R s and R t , i.e.,
Revised: We take all validation images such that the plain processing output x generates the same wrong prediction when fed to R s and R t , i.e.,

Original: R s ( x ) =
Revised: R s ( x ) =

Original: R t ( x ) (cid:54) =
Revised: R t ( x ) (cid:54) =

Original: Ground Truth.
Revised: Ground Truth.

Original: For each image, we compute BD ( R s , x, d ) , BD ( R t , x, d ) and IBD ( R s , R t , x, d ) with d being random direction and RA direction.
Revised: For each image, we compute BD ( R s , x, d ) , BD ( R t , x, d ) and IBD ( R s , R t , x, d ) with d being random direction and RA direction.

Original: In this experiment we present results with R s being R18 and R t being R50, as we observe other model pairs produce similar trends.
Revised: In this experiment we present results with R s being R18 and R t being R50, as we observe other model pairs produce similar trends.

Original: The average results are shown in Table 6.
Revised: The average results are shown in Table 6.

Original: We ﬁrst observe that BDs are much smaller alongside the
Revised: We ﬁrst observe that BDs are much smaller alongside the

Original: RA direction than the random direction.
Revised: RA direction than the random direction.

Original: This indicates moving along the RA direction will change the model’s wrong prediction at x faster, possibly to a correct prediction.
Revised: This indicates moving along the RA direction will change the model’s wrong prediction at x faster, possibly to a correct prediction.

Original: More importantly, under either random or RA direction, IBD is always smaller than source/target BDs, which indicates R s and R t ’s boundaries are relatively close, leading to a transfering effect.
Revised: More importantly, under either random or RA direction, IBD is always smaller than source/target BDs, which indicates R s and R t ’s boundaries are relatively close, leading to a transfering effect.

Original: This result in RA direction can explain why RA processing can lead to transferable accuracy gains, since the RA loss brings this direction as the effect on the processing output x .
Revised: This result in RA direction can explain why RA processing can lead to transferable accuracy gains, since the RA loss brings this direction as the effect on the processing output x .

Original: We further visualize decision boundaries in Fig.
Revised: We further visualize decision boundaries in Fig.

Original: 4 with examples (see Appendix H for more).
Revised: 4 with examples (see Appendix I for more).

Original: We use R18 as source and each of the other models as target.
Revised: We use R18 as source and each of the other models as target.

Original: Here we plot d RA as the horizontal and d r as the vertical axis.
Revised: Here we plot d RA as the horizontal and d r as the vertical axis.

Original: The origin represents the plain processing output x , and the color of point ( u, v ) represents the predicted class of the image x + u · d RA + v · d r .
Revised: The origin represents the plain processing output x , and the color of point ( u, v ) represents the predicted class of the image x + u · d RA + v · d r .

Original: From the plot, we can see that different models share similar decision boundaries, and also tend to change to the same prediction once we move from the origin along a direction far enough.
Revised: From the plot, we can see that different models share similar decision boundaries, and also tend to change to the same prediction once we move from the origin along a direction far enough.

Original: In both examples, we do conﬁrm that when we move towards RA direction (towards right at horizontal axis), the ﬁrst color we encounter (green for top, purple for bottom) represents the image’s correct label.
Revised: In both examples, we do conﬁrm that when we move towards RA direction (towards right at horizontal axis), the ﬁrst color we encounter (green for top, purple for bottom) represents the image’s correct label.

Original: This suggests the signal from
Revised: This suggests the signal from

Original: RA loss (RA direction) can correct the wrong prediction with plain processing output ( x at origin), and such correction is transferable given the similar decision boundaries among models.
Revised: RA loss (RA direction) can correct the wrong prediction with plain processing output ( x at origin), and such correction is transferable given the similar decision boundaries among models.

Original: Conclusion.
Revised: Conclusion.

Original: We investigated an important yet largely ignored problem: enhancing the machine interpretability of image processing outputs.
Revised: We investigated an important yet largely ignored problem: enhancing the machine interpretability of image processing outputs.

Original: We found that our simple approaches – optimizing with the additional recognition loss, can signiﬁcantly boost the recognition accuracy with minimal or no loss in image quality.
Revised: We found that our simple approaches – optimizing with the additional recognition loss, can signiﬁcantly boost the recognition accuracy with minimal or no loss in image quality.

Original: Moreover, such gain can transfer to architectures, object categories, vision tasks unseen during training, or even a black-box cloud model, indicating the enhanced interpretability is not speciﬁc to one particular model but generalizable to others.
Revised: Moreover, such gain can transfer to architectures, object categories, vision tasks unseen during training or even a black-box cloud model, indicating the enhanced interpretability is not speciﬁc to one particular model but generalizable to others.

Original: This makes the approaches applicable even when the future downstream recognition models are unknown.
Revised: This makes the approaches applicable even when the future downstream recognition models are unknown.

Original: Finally we analyzed the decision boundary similarities of recognition models to explain this transferability phenomenon.
Revised: Finally we analyzed the decision boundary similarities of recognition models to explain this transferability phenomenon

Original: A PPENDIX T ABLE OF C ONTENTS
Revised: A PPENDIX T ABLE OF C ONTENTS

Original: A Comparison with Alternatives
Revised: A Comparison with Alternatives

Original: B Experimental Details
Revised: B Experimental Details

Original: C Results on More Architectures
Revised: C Results on More Architectures

Original: D Results on Object Detection (Same Architecture)
Revised: D Results on Object Detection (Same Architecture)

Original: E More Results on Transferability
Revised: E More Results on Transferability

Original: E.1 Transferring between Architectures . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.1 Transferring between Architectures . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 Transferring between Recognition Tasks . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.2 Transferring between Recognition Tasks . . . . . . . . . . . . . . . . . . . . . . .

Original: F Results on ImageNet-C
Revised: F Results on ImageNet-C

Revised: G Experiments on Semantic Segmentation

Original: G More Image Quality Visualizations
Revised: H More Image Quality Visualizations

Original: H More Decision Boundary Visualizations
Revised: I More Decision Boundary Visualizations

Original: A C OMPARISON WITH A LTERNATIVES
Revised: A C OMPARISON WITH A LTERNATIVES

Original: We analyze some alternatives to our approaches.
Revised: We analyze some alternatives to our approaches.

Original: Unless otherwise speciﬁed, experiments in this section are conducted using RA processing on super-resolution, with ResNet-18 trained on ImageNet as the recognition model, and λ = 10 − 3 if used.
Revised: Unless otherwise speciﬁed, experiments in this section are conducted using RA processing on super-resolution, with ResNet-18 trained on ImageNet as the recognition model, and λ = 10 − 3 if used.

Original: Under this setting, we achieve 61.8% classiﬁcation accuracy on the output images.
Revised: Under this setting, we achieve 61.8% classiﬁcation accuracy on the output images.

Original: Training/Fine-tuning the Recognition Model.
Revised: Training/Fine-tuning the Recognition Model.

Original: Instead of ﬁxing the recognition model R , we could train/ﬁne-tune it together with the training of image processing model P , to optimize the recognition loss.
Revised: Instead of ﬁxing the recognition model R , we could train/ﬁne-tune it together with the training of image processing model P , to optimize the recognition loss.

Original: Many prior works (Sharma et al., 2018; Bai et al., 2018;
Revised: Many prior works (Sharma et al., 2018; Bai et al., 2018;

Original: Zhang et al., 2018a) do train/ﬁne-tune the recognition model jointly with the image processing model.
Revised: Zhang et al., 2018a) do train/ﬁne-tune the recognition model jointly with the image processing model.

Original: We use SGD with momentum as R ’s optimizer, and the ﬁnal accuracy reaches 63.0%.
Revised: We use SGD with momentum as R ’s optimizer, and the ﬁnal accuracy reaches 63.0%.

Original: However, since we do not ﬁx R , it becomes a model that speciﬁcally recognizes super-resolved images, and we found its performance on original target images drops from 69.8% to 60.5%.
Revised: However, since we do not ﬁx R , it becomes a model that speciﬁcally recognizes super-resolved images, and we found its performance on original target images drops from 69.8% to 60.5%.

Original: Moreover, when transferring the trained P on ResNet-50, the accuracy is 62.4 %, worse than 66.7% when we train with a ﬁxed ResNet-18.
Revised: Moreover, when transferring the trained P on ResNet-50, the accuracy is 62.4 %, worse than 66.7% when we train with a ﬁxed ResNet-18.

Original: This suggests we lose some transferability if we do not ﬁx the recognition model R .
Revised: This suggests we lose some transferability if we do not ﬁx the recognition model R .

Original: Training Recognition Models from Scratch.
Revised: Training Recognition Models from Scratch.

Original: We could ﬁrst train a super-resolution model, and then train R from scratch on the output images.
Revised: We could ﬁrst train a super-resolution model, and then train R from scratch on the output images.

Original: Doing this, we achieve 66.1% accuracy, higher than 61.8% in RA processing.
Revised: Doing this, we achieve 66.1% accuracy, higher than 61.8% in RA processing.

Original: However, R ’s accuracy on original clean images drops from 69.8% to 66.1%.
Revised: However, R ’s accuracy on original clean images drops from 69.8% to 66.1%.

Original: Alternatively, we could train R from scratch on interpolated low-resolution images, in which case we achieve 66.0% on interpolated validation data but only 50.2% on the original data.
Revised: Alternatively, we could train R from scratch on interpolated low-resolution images, in which case we achieve 66.0% on interpolated validation data but only 50.2% on the original data.

Original: In summary, training/ﬁne-tuning R to cater the need of super-resolved or interpolated images can harm its performance on original images, and causes additional overhead in storing models.
Revised: In summary, training/ﬁne-tuning R to cater the need of super-resolved or interpolated images can harm its performance on original images, and causes additional overhead in storing models.

Original: In contrast, RA processing could boost the accuracy of output images with the performance on original images intact.
Revised: In contrast, RA processing could boost the accuracy of output images with the performance on original images intact.

Original: Training without the Image Processing Loss.
Revised: Training without the Image Processing Loss.

Original: It is possible to train the processing model on the recognition loss L recog , without even keeping the original image processing loss L proc (Eqn. 3).
Revised: It is possible to train the processing model on the recognition loss L recog , without even keeping the original image processing loss L proc (Eqn. 3).

Original: This may presumably lead to better recognition performance since the model P can now “focus on” optimizing the recognition loss.
Revised: This may presumably lead to better recognition performance since the model P can now “focus on” optimizing the recognition loss.

Original: However, we found removing the original image processing loss hurts the recognition performance: the accuracy drops from 61.8% to 60.9%; even worse, the
Revised: However, we found removing the original image processing loss hurts the recognition performance: the accuracy drops from 61.8% to 60.9%; even worse, the

Original: SSIM/PSNR metrics drop from 26.69/0.804 to 16.92/0.263, which is reasonable since the image processing loss is not optimized during training.
Revised: SSIM/PSNR metrics drop from 26.69/0.804 to 16.92/0.263, which is reasonable since the image processing loss is not optimized during training.

Original: This suggests the original image processing loss is helpful for the recognition accuracy, since it helps the corrupted image to restore to its original form.
Revised: This suggests the original image processing loss is helpful for the recognition accuracy, since it helps the corrupted image to restore to its original form.

Original: Perceptual/Feature Loss.
Revised: Perceptual/Feature Loss.

Original: Our unsupervised RA method optimizes the recognition model’s output probability distance between processed and target images.
Revised: Our unsupervised RA method optimizes the recognition model’s output probability distance between processed and target images.

Original: This is related to the perceptual loss (also called feature loss) used in Johnson et al.
Revised: This is related to the perceptual loss (also called feature loss) used in Johnson et al.

Original: ; Ledig et al.
Revised: ; Ledig et al.

Original: Perceptual loss optimizes processed and target images’ distance in VGG feature space.
Revised: Perceptual loss optimizes processed and target images’ distance in VGG feature space.

Original: Note that the perceptual loss was originally proposed to improve output’s quality from a human observer’s perspective.
Revised: Note that the perceptual loss was originally proposed to improve output’s quality from a human observer’s perspective.

Original: To compare both methods, we follow Ledig et al.
Revised: To compare both methods, we follow Ledig et al.

Original: (2017) to optimize the perceptual loss from VGG-16.
Revised: (2017) to optimize the perceptual loss from VGG-16.

Original: We ﬁnd perceptual loss yields lower accuracy than unsupervised RA (56.7% vs. 61.0% on the VGG16 recognition model).
Revised: We ﬁnd perceptual loss yields lower accuracy than unsupervised RA (56.7% vs. 61.0% on the VGG16 recognition model).

Original: This could be because using ﬁnal probabilities provides more semantic supervision, while intermediate features improve the outputs from a perceptual perspective.
Revised: This could be because using ﬁnal probabilities provides more semantic supervision, while intermediate features improve the outputs from a perceptual perspective.

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: General Setup.
Revised: General Setup.

Original: We evaluate our proposed methods on three image processing tasks: image superresolution, denoising, and JPEG-deblocking.
Revised: We evaluate our proposed methods on three image processing tasks: image superresolution, denoising, and JPEG-deblocking.

Original: In those tasks, the target images are all the original images from the datasets.
Revised: In those tasks, the target images are all the original images from the datasets.

Original: To obtain the input images, for super-resolution, we use a downsampling scale factor of 4 × ; for denoising, we add Gaussian noise on the images with a standard deviation of 0.1 to obtain the noisy images; for JPEG deblocking, a quality factor of 10 is used to compress the image to JPEG format.
Revised: To obtain the input images, for super-resolution, we use a downsampling scale factor of 4 × ; for denoising, we add Gaussian noise on the images with a standard deviation of 0.1 to obtain the noisy images; for JPEG deblocking, a quality factor of 10 is used to compress the image to JPEG format.

Original: The image processing loss used is the mean squared error (MSE, or L 2 ) loss.
Revised: The image processing loss used is the mean squared error (MSE, or L 2 ) loss.

Original: For the recognition tasks, we consider image classiﬁcation and object detection, two common tasks in computer vision.
Revised: For the recognition tasks, we consider image classiﬁcation and object detection, two common tasks in computer vision.

Original: In total, we have 6 (3 × 2) task pairs to evaluate.
Revised: In total, we have 6 (3 × 2) task pairs to evaluate.

Original: Training is performed with the training set and results on the validation set are reported.
Revised: Training is performed with the training set and results on the validation set are reported.

Original: We adopt the SRResNet (Ledig et al., 2017) as the architecture of the image processing model P , which is simple yet effective in optimizing the MSE loss.
Revised: We adopt the SRResNet (Ledig et al., 2017) as the architecture of the image processing model P , which is simple yet effective in optimizing the MSE loss.

Original: Even though SRResNet is originally designed for super-resolution, we ﬁnd it also performs well on denoising and JPEG deblocking when its upscale parameter set to 1 for the same input-output sizes.
Revised: Even though SRResNet is originally designed for super-resolution, we ﬁnd it also performs well on denoising and JPEG deblocking when its upscale parameter set to 1 for the same input-output sizes.

Original: Throughout the experiments, on both the image processing network and the transformer, we use the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 10 − 4 , following the original SRResNet
Revised: Throughout the experiments, on both the image processing network and the transformer, we use the Adam optimizer (Kingma & Ba, 2014) with an initial learning rate of 10 − 4 , following the original SRResNet

Original: (Ledig et al., 2017).
Revised: (Ledig et al., 2017).

Original: Our implementation is in PyTorch (Paszke et al., 2017).
Revised: Our implementation is in PyTorch (Paszke et al., 2017).

Original: Each result is based on one run due to the relatively stable performance (almost all within 1%) we noticed and the large amount of tasks combinations/architectures to be evaluated.
Revised: Each result is based on one run due to the relatively stable performance (almost all within 1%) we noticed and the large amount of tasks combinations/architectures to be evaluated.

Original: For example, on super-resolution with ResNetas R on ImageNet, ﬁve runs on plain processing gives accuracies (%): 53.9, 53.8, 54.0, 53.8, 53.
Revised: For example, on super-resolution with ResNetas R on ImageNet, ﬁve runs on plain processing gives accuracies (%): 53.9, 53.8, 54.0, 53.8, 53.

Original: (53.88 ± 0.07); RA processing: 61.7, 61.9, 61.6, 61.8, 61.7 (61.74 ± 0.10); unsupervised RA: 61.2, 61.3, 61.3, 61.4, 61.1 (61.26 ± 0.10); RA w/ Transformer: 62.9, 63.0, 62.9, 62.9, 63.0 (62.94 ± 0.05).
Revised: (53.88 ± 0.07); RA processing: 61.7, 61.9, 61.6, 61.8, 61.7 (61.74 ± 0.10); unsupervised RA: 61.2, 61.3, 61.3, 61.4, 61.1 (61.26 ± 0.10); RA w/ Transformer: 62.9, 63.0, 62.9, 62.9, 63.0 (62.94 ± 0.05).

Original: The experiments are run on 1-4 NVIDIA TITAN
Revised: The experiments are run on 1-4 NVIDIA TITAN

Original: Xp GPUs.
Revised: Xp GPUs.

Original: Clearly training time is linear in the iterations trained and the space taken is a constant with a ﬁxed input size.
Revised: Clearly training time is linear in the iterations trained and the space taken is a constant with a ﬁxed input size.

Original: The training process ﬁnishes in 2-24 hours depending on the model sizes/variants of methods/recognition tasks, and the maximum GPU memory taken is 30GB (multi-GPU) with batch size of 20.
Revised: The training process ﬁnishes in 2-24 hours depending on the model sizes/variants of methods/recognition tasks, and the maximum GPU memory taken is 30GB (multi-GPU) with batch size of 20.

Original: Image Classiﬁcation.
Revised: Image Classiﬁcation.

Original: For image classiﬁcation, we evaluate our method on the large-scale ImageNet benchmark (Deng et al., 2009), which can be downloaded at http://image-net.org/ download .
Revised: For image classiﬁcation, we evaluate our method on the large-scale ImageNet benchmark (Deng et al., 2009), which can be downloaded at http://image-net.org/ download .

Original: It consists of ∼ 1 .
Revised: It consists of ∼ 1 .

Original: 2 million training images and 50,000 validation images.
Revised: 2 million training images and 50,000 validation images.

Original: We use ﬁve pre-trained image classiﬁcation models, ResNet-18/50/101 (He et al., 2016), DenseNet-121 (Huang et al., 2017) and VGG-16 (Simonyan & Zisserman, 2015) with BN (Ioffe & Szegedy, 2015) (denoted as R18/50/101, D121, V16 in Table 1), on which the top-1 accuracy (%) of the original validation images is 69.8, 76.2, 77.4, 74.7, and 73.4 respectively.
Revised: We use ﬁve pre-trained image classiﬁcation models, ResNet-18/50/101 (He et al., 2016), DenseNet-121 (Huang et al., 2017) and VGG-16 (Simonyan & Zisserman, 2015) with BN (Ioffe & Szegedy, 2015) (denoted as R18/50/101, D121, V16 in Table 1), on which the top-1 accuracy (%) of the original validation images is 69.8, 76.2, 77.4, 74.7, and 73.4 respectively.

Original: We train the processing models for 6 epochs on the training set, with a learning rate decay of 10 × at epoch 5 and 6, and a batch size of 20.
Revised: We train the processing models for 6 epochs on the training set, with a learning rate decay of 10 × at epoch 5 and 6, and a batch size of 20.

Original: In evaluation, we feed unprocessed validation images to the image processing model, and report the accuracy of the output images evaluated on the pre-trained classiﬁcation networks.
Revised: In evaluation, we feed unprocessed validation images to the image processing model, and report the accuracy of the output images evaluated on the pre-trained classiﬁcation networks.

Original: For unsupervised
Revised: For unsupervised

Original: RA, we use L 2 distance as the function l dis in Eqn. 4.
Revised: RA, we use L 2 distance as the function l dis in Eqn. 4.

Original: The hyperparameter λ is chosen using super-resolution with the ResNet-18 recognition model, on two small subsets for training/validation from the original large training set, from a grid search from 10 − 4 to 100.
Revised: The hyperparameter λ is chosen using super-resolution with the ResNet-18 recognition model, on two small subsets for training/validation from the original large training set, from a grid search from 10 − 4 to 100.

Original: The λ chosen for RA processing, RA with transformer, and unsupervised RA is 10 − 3 , 10 − 2 and 10 respectively.
Revised: The λ chosen for RA processing, RA with transformer, and unsupervised RA is 10 − 3 , 10 − 2 and 10 respectively.

Original: Object Detection.
Revised: Object Detection.

Original: For object detection, we evaluate on PASCAL VOC 2007 and 2012 detection dataset ( https://pjreddie.com/projects/pascal-voc-dataset-mirror/ ), using Faster-RCNN (Ren et al., 2015) as the recognition model.
Revised: For object detection, we evaluate on PASCAL VOC 2007 and 2012 detection dataset ( https://pjreddie.com/projects/pascal-voc-dataset-mirror/ ), using Faster-RCNN (Ren et al., 2015) as the recognition model.

Original: Our implementation is based on the code from (Yang et al., 2017).
Revised: Our implementation is based on the code from (Yang et al., 2017).

Original: Following common practice (Redmon et al., 2016; Ren et al., 2015;
Revised: Following common practice (Redmon et al., 2016; Ren et al., 2015;

Original: Dai et al., 2016), we use VOC 07 and 12 trainval data as the training set, and evaluate on VOCtest data.
Revised: Dai et al., 2016), we use VOC 07 and 12 trainval data as the training set, and evaluate on VOCtest data.

Original: The Faster-RCNN training uses the same hyperparameters in (Yang et al., 2017).
Revised: The Faster-RCNN training uses the same hyperparameters in (Yang et al., 2017).

Original: For the recognition model’s backbone architecture, we evaluate ResNet-18/50/101 and VGG-16 (without
Revised: For the recognition model’s backbone architecture, we evaluate ResNet-18/50/101 and VGG-16 (without

Original: BN (Ioffe & Szegedy, 2015)), obtaining mAP of 74.2, 76.8, 77.9, 72.2 on the test set respectively.
Revised: BN (Ioffe & Szegedy, 2015)), obtaining mAP of 74.2, 76.8, 77.9, 72.2 on the test set respectively.

Original: Given those trained detectors as recognition loss functions, we train the models on the training set for 7 epochs, with a learning rate decay of 10 × at epoch 6 and 7, and a batch size of 1.
Revised: Given those trained detectors as recognition loss functions, we train the models on the training set for 7 epochs, with a learning rate decay of 10 × at epoch 6 and 7, and a batch size of 1.

Original: We report the mean Average Precision (mAP) of processed images in the test set.
Revised: We report the mean Average Precision (mAP) of processed images in the test set.

Original: As in image classiﬁcation, we use λ = 10 − 3 for RA processing, and λ = 10 − 2 for RA with transformer.
Revised: As in image classiﬁcation, we use λ = 10 − 3 for RA processing, and λ = 10 − 2 for RA with transformer.

Original: C R ESULTS ON M ORE A RCHITECTURES
Revised: C R ESULTS ON M ORE A RCHITECTURES

Original: In our main paper, we use SRResNet (Ledig et al., 2017) as our processing model P .
Revised: In our main paper, we use SRResNet (Ledig et al., 2017) as our processing model P .

Original: Here we provide more results with other more recent processing models, including SRDenseNet (SRDNet) (Tong et al., 2017), Residual Dense Network (RDN)
Revised: Here we provide more results with other more recent processing models, including SRDenseNet (SRDNet) (Tong et al., 2017), Residual Dense Network (RDN)

Original: (Zhang et al., 2018d), and Deep Back-Projection Networks (DBPN)
Revised: (Zhang et al., 2018d), and Deep Back-Projection Networks (DBPN)

Original: Haris et al.
Revised: Haris et al.

Original: (2018b). We present results at Table 7, with super-resolution as the processing task, ImageNet classiﬁcation as recognition task, and R being ResNet-18. 
Revised: We present results at Table 7, with super-resolution as the processing task, ImageNet classiﬁcation as recognition task, and R being ResNet-18. is supre-resolution.

Original: [Table]
Revised: [Table]

Original: D R ESULTS ON O BJECT D ETECTION (S AME A RCHITECTURE )
Revised: D R ESULTS ON O BJECT D ETECTION (S AME A RCHITECTURE )

Original: The results for object detection, when evaluated on the same architecture, are shown in Table 8.
Revised: The results for object detection, when evaluated on the same architecture, are shown in Table 8.

Original: We observe similar trend as in classiﬁcation: using recognition loss can consistently improve the mAP over plain image processing by a notable margin.
Revised: We observe similar trend as in classiﬁcation: using recognition loss can consistently improve the mAP over plain image processing by a notable margin.

Original: On super-resolution, RA processing mostly performs on par with RA with transformer, but on the other two tasks using a transformer is slightly better.
Revised: On super-resolution, RA processing mostly performs on par with RA with transformer, but on the other two tasks using a transformer is slightly better.

Original: The model with transformer performs better more often possibly because with this extra network in the middle, the capacity of the whole system is increased.
Revised: The model with transformer performs better more often possibly because with this extra network in the middle, the capacity of the whole system is increased.

Original: [Table]
Revised: [Table]

Original: E M ORE R ESULTS ON T RANSFERABILITY
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: {+RA w/ Transformer 71.
+}
Revised: {+RA w/ Transformer 71.
+}

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: E

Revised: M

Revised: ORE R ESULTS ON T RANSFERABILITY

Original: We present additional results on transferability in this section. 
Revised: We present additional results on transferability in this section. [Table] [Table]

Original: E. T RANSFERRING BETWEEN A RCHITECTURES
Revised: T RANSFERRING BETWEEN A RCHITECTURES

Original: We provide the model transferability results of RA processing on object detection in Table 9.
Revised: We provide the model transferability results of RA processing on object detection in Table 9.

Original: Rows indicate the models trained as recognition loss and columns indicate the evaluation models.
Revised: Rows indicate the models trained as recognition loss and columns indicate the evaluation models.

Original: We[Table] see similar trend as in classiﬁcation (Table 1): using other architectures as loss can also improve recognition performance over plain processing; the loss model that achieves the highest performance is mostly the model itself, as can be seen from the fact that most boldface numbers are on the diagonals.
Revised: We see similar trend as in classiﬁcation (Table 1): using other architectures as loss can also improve recognition performance over plain processing; the loss model that achieves the highest performance is mostly the model itself, as can be seen from the fact that most boldface numbers are on the diagonals.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we present the results when transferring between recognition architectures, using unsupervised RA.
Revised: we present the results when transferring between recognition architectures, using unsupervised RA.

Original: We note that for super-resolution and JPEG-deblocking, similar trend holds as in (supervised) RA processing, as using any architecture in training will improve over plain processing.
Revised: We note that for super-resolution and JPEG-deblocking, similar trend holds as in (supervised) RA processing, as using any architecture in training will improve over plain processing.

Original: But for denoising, this is not always the case.
Revised: But for denoising, this is not always the case.

Original: Some models P trained with unsupervised RA are slightly worse than the plain processing counterpart.
Revised: Some models P trained with unsupervised RA are slightly worse than the plain processing counterpart.

Original: A possible reason for this is the noise level in our experiments is not large enough and plain processing achieve very high accuracy already.
Revised: A possible reason for this is the noise level in our experiments is not large enough and plain processing achieve very high accuracy already.

Original: [Table]
Revised: [Table]

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we present the results of transferring between architectures when we use a transformer
Revised: we present the results of transferring between architectures when we use a transformer

Original: T .
Revised: T .

Original: We use the processing model P and transformer T trained with R A together when evaluating on
Revised: We use the processing model P and transformer T trained with R A together when evaluating on

Original: R B .
Revised: R B .

Original: From Table [Table] in most cases improvement is still transferable but there are a few exceptions.
Revised: From Table 11, in most cases improvement is still transferable but there are a few exceptions.

Original: For example, when R A is ResNet or DenseNet and when R B is VGG-16, in most cases the accuracy fall behind plain processing by a large margin.
Revised: For example, when R A is ResNet or DenseNet and when R B is VGG-16, in most cases the accuracy fall behind plain processing by a large margin.

Original: This weaker transferability is possibly caused by the fact that there is no constraint imposed by the image processing loss on T ’s output, thus it “overﬁts” more to the speciﬁc R it is trained with.
Revised: This weaker transferability is possibly caused by the fact that there is no constraint imposed by the image processing loss on T ’s output, thus it “overﬁts” more to the speciﬁc R it is trained with.

Original: [Table]
Revised: [Table]

Original: E. T RANSFERRING BETWEEN R ECOGNITION T ASKS
Revised: E. T RANSFERRING BETWEEN R ECOGNITION T ASKS

Original: In Section
Revised: In Section

Original: [Table]
Revised: [Table]

Original: we investigated the transferability of improvement from classiﬁcation to detection.
Revised: we investigated the transferability of improvement from classiﬁcation to detection.

Original: Here we evaluate the opposite direction, from detection to classiﬁcation.
Revised: Here we evaluate the opposite direction, from detection to classiﬁcation.

Original: The results are shown in Table 12.
Revised: The results are shown in Table 12.

Original: Here, using RA processing can still consistently improve over plain processing for any pair of models, but we note that the improvement is not as signiﬁcant as directly training using classiﬁcation models as loss (Table 1 and Table 2).
Revised: Here, using RA processing can still consistently improve over plain processing for any pair of models, but we note that the improvement is not as signiﬁcant as directly training using classiﬁcation models as loss (Table 1 and Table 2).

Original: [Table]
Revised: [Table]

Original: Additionally, the results when we transfer the model P trained with unsupervised RA with image classiﬁcation to object detection are shown in Table 13.
Revised: Additionally, the results when we transfer the model P trained with unsupervised RA with image classiﬁcation to object detection are shown in Table 13.

Original: In most cases, it improves over plain processing, but for image denoising, this is not always the case.
Revised: In most cases, it improves over plain processing, but for image denoising, this is not always the case.

Original: Similar to results in Table 10, this could be because the noise level is relatively low in our experiments.
Revised: Similar to results in Table 10, this could be because the noise level is relatively low in our experiments.

Original: [Table]
Revised: [Table]

Original: F R ESULTS ON I MAGE N ET -C
Revised: F R ESULTS ON I MAGE N ET -C

Original: We evaluate our methods on the ImageNet-C benchmark (Hendrycks & Dietterich, 2019).
Revised: We evaluate our methods on the ImageNet-C benchmark (Hendrycks & Dietterich, 2019).

Original: It imposes 17 different types of corruptions on the ImageNet (Deng et al., 2009) validation set.
Revised: It imposes 17 different types of corruptions on the ImageNet (Deng et al., 2009) validation set.

Original: Despite ImageNetC benchmark is designed for more robust recognition models, but not for testing image processing models, it is a good testbed to test our methods in a broader range of processing tasks.
Revised: Despite ImageNetC benchmark is designed for more robust recognition models, but not for testing image processing models, it is a good testbed to test our methods in a broader range of processing tasks.

Original: Since only corrupted images from the validation set are released, we divide it evenly for each class into two halves and train/test on its ﬁrst/second half.
Revised: Since only corrupted images from the validation set are released, we divide it evenly for each class into two halves and train/test on its ﬁrst/second half.

Original: The corrupted image is the input image to the processing model and the original clean image is the target image.
Revised: The corrupted image is the input image to the processing model and the original clean image is the target image.

Original: The recognition model used in this experiment is an ImageNet-pretrained ResNet-18.
Revised: The recognition model used in this experiment is an ImageNet-pretrained ResNet-18.

Original: [Table]
Revised: [Table]

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we evaluate RA Processing on all 17 types of corruptions, with corruption level 5 as in (Hendrycks & Dietterich, 2019).
Revised: we evaluate RA Processing on all 17 types of corruptions, with corruption level 5 as in (Hendrycks & Dietterich, 2019).

Original: We observe that RA Processing brings consistent improvement over plain processing, sometimes by an even larger margin than the tasks considered in Sec. 4.
Revised: We observe that RA Processing brings consistent improvement over plain processing, sometimes by an even larger margin than the tasks considered in Sec. 4.

Original: [Table]
Revised: [Table]

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we experiment with different levels of corruptions with corruption type “speckle noise” and “snow”.
Revised: we experiment with different levels of corruptions with corruption type “speckle noise” and “snow”.

Original: We also evaluate with our variants – Unsupervised RA and RA with Transformer.
Revised: We also evaluate with our variants – Unsupervised RA and RA with Transformer.

Original: We observe that when the corruption level is higher, our methods tend to bring more recognition accuracy gain.
Revised: We observe that when the corruption level is higher, our methods tend to bring more recognition accuracy gain.

Original: In this case [Table] we note that using a Transformer could sometimes hurt the accuracy compared with plain processing.
Revised: In this case, we note that using a Transformer could sometimes hurt the accuracy compared with plain processing.

Original: This is possibly because the insufﬁcient training data in ImageNet-C dataset caused the transformer to hurt, since more parameters typically require more training data.
Revised: This is possibly because the insufﬁcient training data in ImageNet-C dataset caused the transformer to hurt, since more parameters typically require more training data.

Original: In the majority of other cases, it improves slightly over RA processing
Revised: In the majority of other cases, it improves slightly over RA processing

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In Table 16, we examine the transferability of RA Processing between recognition architectures, using the same two tasks “speckle noise” and “snow”, with corruption level 5.
Revised: In Table 16, we examine the transferability of RA Processing between recognition architectures, using the same two tasks “speckle noise” and “snow”, with corruption level 5.

Original: Note the recognition loss used during training is from a ResNet-18, and we evaluate the improvement over plain processing on
Revised: Note the recognition loss used during training is from a ResNet-18, and we evaluate the improvement over plain processing on

Original: ResNet-50/101, DenseNet-121 and VGG-16.
Revised: ResNet-50/101, DenseNet-121 and VGG-16.

Original: We observe that the improvement over plain processing is transferable among different architectures.
Revised: We observe that the improvement over plain processing is transferable among different architectures.

Original: G M ORE I MAGE Q UALITY V ISUALIZATIONS
Revised: G E XPERIMENTS ON S EMANTIC S EGMENTATION

Revised: We experiment with the Cityscapes (Cordts et al., 2016) semantic segmentation task using the recent

Revised: HRNet-W48 Wang et al.

Revised: (2020) architecture as the recognition model.

Revised: As with other tasks, we use a λ of 0.001 for RA processing, and SRResNet as processing model.

Revised: We train for 18 epochs and other settings are the same as in the main paper.

Revised: [Table]

Revised: Table 17 compares the recognition accuarcy (mIoU) of Plain and RA models.

Revised: We observe that RA processing is able to improve the acuracy substantially, with similar image qualities in terms of

Revised: PSNR/SSIM.

Revised: [Table]

Revised: We also evaluate the transferring effect from ImageNet classiﬁcation to Cityscapes semantic segmentation.

Revised: We take a plain processing model and an RA processing model (ResNet-18 as recognition model) trained with ImageNet classiﬁcation, and compare their outputs’ performance on Cityscapes segmentation.

Revised: Table 18 shows, using RA Processing on ImageNet classiﬁcation can help segmentation accuracy on Cityscapes as well.

Revised: We also note both accuracies are higher than when we train the processing model on Cityscapes.

Revised: This is possibly due to the abundance of data in ImageNet (1. million images) compared with Cityscapes (3k images).

Revised: H M ORE I MAGE Q UALITY V ISUALIZATIONS

Original: We provide more visualizations in Fig.
Revised: We provide more visualizations in Fig.

Original: 5 where the output image is incorrectly classiﬁed by ResNet18 with a plain image processing model, and correctly recognized with RA processing, as in Fig.
Revised: 5 where the output image is incorrectly classiﬁed by ResNet18 with a plain image processing model, and correctly recognized with RA processing, as in Fig.

Original: 3 at
Revised: 3 at

Original: Section 5.
Revised: Section 5.

Original: [Figure]
Revised: [Figure]

Original: H M ORE D ECISION B OUNDARY V ISUALIZATIONS
Revised: I M ORE D ECISION B OUNDARY V ISUALIZATIONS

Original: We provide more visualizations in Fig.
Revised: We provide more visualizations in Fig.

Original: 6 of different model’s decision boundaries to complement Fig.
Revised: 6 of different model’s decision boundaries to complement Fig.

Original: We can see that different recognition models share similar boundaries.
Revised: We can see that different recognition models share similar boundaries.