Rethinking Image Restoration for Object Detection
Abstract
Although image restoration has achieved significant progress, its potential to assist object detectors in adverse imaging conditions lacks enough attention in the research community.
It is reported that the existing image restoration methods cannot improve the object detector performance and sometimes even reduce the detection performance.
To address the issue, we propose a targeted adversarial attack in the restoration procedure to boost object detection performance after restoration.
Specifically, we present an ADAM-like adversarial attack to generate pseudo ground truth for restoration fine-tuning.
Resultant restored images are close to original sharp images, and at the same time, lead to better object detection results.
We conduct extensive experiments in image dehazing and low light enhancement and show the superiority of our method over conventional training and other domain adaptation and multi-task methods.
The proposed pipeline can be applied to all restoration methods and both one- and two-stage detectors.
Introduction
Image quality degradation is common in outdoor captured images and many image restoration methods have been developed to automatically recover sharp details from the degraded images.
Typically, the goal of image restoration is to generate sharper results from which human vision can better identify the structural details and objects in the images.
However, in many applications, large-scale images are collected for high-level computer vision tasks ( e g. , object detection, semantic segmentation, and autonomous driving) other than a visual examination by human.
Though significant progress has been made in terms of the visual quality, its ability to serve as a pre-processing step for downstream vision tasks lacks enough research attention.
Object detection has achieved remarkable success as the boosting of deep neural networks.
Many successful models, e.g. , Faster-RCNN [32] and YOLOv3 [31], have been developed and deployed in real-world applications such as autonomous driving [4, 38].
However, in the settings of the commonly used benchmark for object detection, poor weather and imaging conditions are intentionally avoided [7] or insufficient due to data bias [16, 9].
The unbalanced data distribution toward clean images leads to a domain shift between properly captured images low-quality ones adverse environments [36].
The performance of object detector trained on high-quality samples thus downgrades significantly in adverse imaging and weather conditions such as haze, rain, and low light exposure [23, 5] since the weather-related deterioration features distort captured image [29, 47].
Many existing solutions to the issue lie in domain adaptation from clean images (source domain) to corrupted ones (target domain) [5, 13, 36].
They assume there exists a distribution shift between clean images and those in poor conditions, and adapt detectors by unsupervised weather-specific priors or domain adaptation components to align the features of two distributions to reduce domain discrepancy.
However, the detector after adaptation is found to perform worse in the source domain than that without adaptation [23] since adapted detectors may focus on weather-relevant deterioration information.
Another track to tackle the issue is to jointly train a simple visual quality enhancement module together with detection network [5, 23].
However, these modules cannot provide satisfactory results compared to powerful restoration methods.
Many artifacts and noise patterns occur when applying the enhancement module.
It is also challenging to balance visual quality and detection performance by tuning hyper-parameters during training.
Since the detector is trained on corrupted samples or the combination of corrupted and clean samples, its performance on clean samples may decrease, similar to the case of domain adaptation.
Different from previous works modifying object detectors, we address the issue by adapting restoration algorithms such that recovered images can achieve both good visual quality and better detection results for detectors.
Though image restoration methods produce satisfying results from the perspective of human eyes, detectors may consider them as variables following the distribution of an artificial neural network rather than the real-world, thus being confused by such a covariate shift.
Therefore, finding a pseudo ground truth close to the original sharp image potentially improves detection.
To find such pseudo ground truth, a targeted adversarial attack can be employed to add an invisibly small perturbation to the original ground truth, on which the detector will give our desired prediction.
The resultant adversarial examples are obtained by minimizing the loss of object detectors over perturbation, and can serve as pseudo ground truth for restoration algorithms.
Such perturbation is trivial, so restoration models are not impacted to the greatest extent.
During the fine-tuning stage, only the weights of restoration methods are updated and detectors keep unchanged.
Thus the performance of detectors will not decline in the clean domain.
Note that the goal of this paper is not the development of a new image restoration algorithm or a new object detection model.
Instead, we propose a targeted attack preprocessing of image ground truth to help improve the accuracy of downstream object detection.
Specifically, we propose a momentum-based Adam [17]-variant adversarial example generation method to estimate pseudo ground truth.
Our contributions can be summarized as following:
We propose a fine-tuning pipeline to improve detection performance and retain visually satisfactory restoration results.
Specifically, we show multi-task optimization problem of balancing both detection and restoration can be decomposed into a targeted adversarial attack generation and a restoration problem.
The pseudo ground truth generated by adversarial example generation is used for fine-tuning.
We design a momentum-based ADAM-like iterative targeted adversarial example generation algorithm to estimate pseudo ground truth for restoration models fine-tuning.
Rather than updating samples by first-order gradient in existing work, we compute ADAM-variant clipped and signed gradient with momentum estimation.
We conduct extensive experiments in two image restoration tasks, i e. , dehazing and low light enhancement for both one- and two-stage object detectors.
We first evaluate the effect of our adversarial attack algorithm compared to existing ones on test sets.
We then show our proposed fine-tuning pipeline effectively improves the detection performance gain of restoration algorithms as a pre-processing step of detectors compared to conventional training and other existing works.
Related Works 2.1 Restoration for Detection
In the image restoration task, many works have achieved promising performance in terms of visual quality and quantitative evaluations [24, 8, 20, 35, 33].
Though many of them mention their potential [Equation] usage as a pre-processing for downstream high-level tasks ( e.g. , object detection and segmentation), only a few of them experiments on how much gain of detection accuracy can be acquired before and after the dehazing step.
Dong et al.
[8] simulate haze on KITTI [10] and show their method removes the synthetic haze effectively and can improve YOLOv3 [31] detection predictions.
Li et al.
[20] publicize the RESIDE dataset, which contains a subset in real-world hazy environment for evaluating object detection on dehazed results.
A recent work [23] appends a differentiable image processing module consisting of multiple traditional filters on YOLOv3, and trains the module and YOLOv together on the combination of clean and hazy samples.
Due to the limitation of filters, it generates many noises and artifacts and thus returns many false positive detection predictions.
In low-light environments, several object detection and semantic segmentation datasets have been collected.
Yang et al.
[46] create a benchmark for face detection in real-world low light condition.
Loh et al.
[26] publish a object detection benchmark in low light environment.
Recently, Sakaridis et al.
[34] collect a comprehensive semantic segmentation dataset in dark imaging condition.
To tackle low-light enhancement task, ZeroDCE [12] is developed by formulating light enhancement as a task of image-specific curve estimation in the pixel-wise range.
Its performance of helping face detection in low-light conditions is tested on DarkFace [46].
Ma et al.
[27] recently develop a new Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brightening images in real-world low-light scenarios.
They show its improvement for low-light face detection and segmentation tasks on ACDC [34] and DarkFace [46] datasets.
The above-mentioned works either merely experiment on synthetic datasets or have to tune detection models together with a restoration module, which limits the generalizability and extensibility of their works on real-world benchmarks and various tasks.
In this work, we focus on proposing a plug-and-play targeted attack module for restoration algorithms to improve the performance of object detection without tuning detection models.
2.2 Adversarial Attack on Detector
After Szegedy et al.
[37] finds that adding an imperceptible perturbation to an image may fool neural network classifier, many adversarial attack generation methods are presented in classification [11, 30, 28, 3, 18].
Fewer works about adversarial attack on object detector are proposed [25] because of its more complicated architecture and loss terms [41, 2].
Some works succeed in fooling detectors by either wearable patch [40, 45, 42] or non-planar patch [50, 48, 43, 14, 39].
Only limited works are about targeted attack.
Dense Adversary Generation (DAG)
[44], Robust Adversarial Perturbation (RAP)
[22], and Contextual Adversarial Perturbation (CAP) [49] are designed for the two-stage detectors consisting of region proposal network.
Other works such as Unified and Efficient Adversary (UEA) [41] and Targeted adversarial Objectness Gradient attack (TOG) [6] can attack both one-stage and two-stage detectors with more generic settings.
UEA is a Generative Adversarial Network framework trained with class loss and feature loss that generates transferable adversarial images and videos.
TOG [6] consists of three relatively powerful adversarial attack algorithms for objectvanishing, object-fabrication, and object-mislabeling attacks, respectively.
A recent work [2] develops a content-aware blackbox adversarial attack framework based on TOG.
In our work, we use TOG as the baseline for whitebox targeted adversarial example generation due to its flexibility and effectiveness in conducting extensive experiments.
We assume detectors are whitebox because our goal is to improve its performance rather than fool it.
Proposed Method 3.1 Pseudo Ground Truth Generation for Restoration
Denote a restoration model with parameters ω as R and a detector with parameter θ as D .
Suppose a dataset for training contains three components: corrupted images x , corresponding clean images ˆx , and detection annotations ˆy .
Given that D only learns on clean samples { ˆx , ˆy } , the prediction of the detector on corrupted samples D ( x ) is probably incorrect.
Our aim is to fine-tune a restorer R such that the recovered image is close to clean image, i.e. , R ( x ) ≈ ˆx , and also the detector give correct prediction on the recovered image, i.e. , D ( R ( x ))
= ˆy .
[Figure]
Traditional supervised restoration problem is not involved with the detection objective and can be generally formulated as:
[Equation]
However, solely training with the objective 1 cannot guarantee a satisfactory result in subsequent object detection since the covariate shift and artifacts as reported in [20, 23, 21].
Given that restoration is an ill-posed problem, various possible recovered images exist for a degraded one.
Considering the inevitable image noises due to hardware limitations during the image acquisition and transmission stage especially in adverse weather and imaging conditions, captured clean images can unnecessarily be regarded as a perfect ground truth.
Unlike conventional restoration methods, our goal is to find a balance between visual quality and detection performance on restored images.
Our objective is to make detector D inference correctly on the restored image as close as possible to the clean image.
Thus it can be considered as a multi-task optimization problem by combining the restoration constraint and object detection loss.
We start by formulating it as follows,
[Equation]
where ˜x = R ( x ) is the recovered image, s is a slackness coefficient indicating the allowed distance from the ground truth ˆx .
The constraint optimization problem cannot be solved easily due to the constraint.
Thus we formulate its dual function as below by introducing a dual variable α ≥ 0 as,
[Equation]
Theorem 1 For any s > 0 , there exists α ≥ 0 such that the optimal solution of (2), ˜x ∗ , is also optimal for (3) , and vise versa.
The proof of Theorem 1 is attached in the supplemental materials.
The main idea is to use Lagrangian and a strong duality assumption.
By Theorem 1, we can say the original problem Eq.2 and the dual problem is equivalent.
However, the multi-task training faces difficulty in tuning the hyper-parameter α , and an improper α leads to either poor detection or restoration results.
Figure 1 shows a case where optimizing d 1 cannot ensure correct detection.
Thus we further reformulate the objective and bypass the issue.
By substituting β = 1 α , we get
[Equation]
T , Update stepsize λ , Magnitude
Perturbation δ
Output: Attacked adversarial examples
∗ ,
Assign β 1 = 0 .
9 , β 2 = 0 .
999 , ϵ = 1 e −
Initialize m 0 = 0 , v 0 = 0 , z = rand ( − δ, δ ) ; x ← clip ( x + z , 0 , 1) foreach ( x , ˆy ) in U do foreach t in 1 ... T do g x ← ∇ x L θ ( D ( x ) , ˆy ) m
[Equation]
v
[Equation]
+ (1 − β 2 ) g 2 x ˆm ← m t / (1 − β t 1 ) ˆv ← v t / (1 − β t 2 ) Update x λ
+ )] ; z ) x
Store
By Theorem 1 again, we have
[Equation]
where r is a small slackness coefficient that regularizes the detection performance.
By introducing an intermediate variable ˆx ∗ that is close enough to ˆx , we have the expression of optimization, min ˜x ∥ ˆx ∗ − ˜x ∥ , where ˆx ∗ =
arg min x
[Equation]
The solutions of (5) and (6) are equivalent when ˜x ∗ =
ˆx ∗ .
The problem can thus be separated into two optimization steps,
[Equation]
Therefore, (7) is an objective of targeted adversarial attack if we substitute for x = ˆx + z and add box constraint ∥ x ∥ ≤ 1 .
Then, (7) can yield a detector-friendly sample close to the original ground truth, and (8) forces restoration models to learn the pattern mapping corrupted samples to the pseudo ground truth.
The solution of (5) can be approximately computed by the fine-tuning pipeline of sequentially optimizing (7) and (8).
3.2 Targeted Adversarial Attack on Object Detector
Object detector D receives an input image x and predicts label with n detected objects, i.e. , y = { ( b x , b y , b H , b W , C, p ) i | i
= 1 ...n } , where ( b x , b y , b H , b W ) represents the location and region of bounding box for i -th object, C is classification result and p is prediction confidence.
The existing method, TOG [6] designs a group of loss terms specific for object detector.
The adversarial example is updated iteratively by signed and clipped first-order gradient of the loss term as,
[Equation]
(9) where λ is the step size of perturbation update and δ is the allowed perturbation scale centered at original input x .
However, we note that the first-order gradient descent used in TOG may get stuck at sub-optimal or converge slowly.
Therefore, we introduce the momentum-based optimizer
ADAM [17] to optimize adversarial examples during updating perturbation, which is widely used in optimizing parameters of neural networks during training.
It can adaptively adjust learning rate by second-order momentum and overcome sub-optimal with cumulative momentum.
Therefore, we reformulate (9) into
[Equation]
Finally, the generated pseudo ground truth ˆx ∗ are used to fine-tune restoration models.
The main step of the targeted adversarial attack is presented in Algorithm 1.
The convergence analysis of the algorithm is discussed in the supplemental material.
Experiments 4.1 Experimental Settings
We conduct experiments on two image restoration tasks, i.e. , image dehazing and low-light image enhancement, and two detectors of YOLOv3 [31] and Faster-RCNN [32].
Dehazing Dataset: RTTS [20] is a real-world hazy dataset with detection annotation for test purpose.
The object labels contain five categories, i.e. , person, car, bus, bicycle, and motorcycle .
To construct a training set with both ground truth clean image and bounding box annotation, we select those PASCAL
VOC images with the above-mentioned five classes and simulate haze on them following [23].
We only simulate two scales of haze rather than ten in [23], and thus our training set is only 20% of theirs.
RTTS contains 4,322 real-world hazy images.
VOC_fog_train and VOC_fog_test consist of 16, and 2,734 synthetic hazy images respectively.
Low-light Enhancement Dataset: ExDark [26] is a natural low light dataset for object detection consisting of ten classes, i.e. , bicycle, boat, bottle, bus, car, cat, chair, dog, motorbike, person .
Similar to dehazing task, we select those samples in PASCAL VOC with these ten categories and simulate low light corruption to form the training set, dubbed as VOC_dark_train.
ExDark contains 2, real-world dark images.
VOC_dark_train and VOC_dark_test consist of 12,334 and 3,760 synthetic dark images respectively.
Metrics: We adopt mean average precision (mAP) with an IOU threshold 50% as a metric for detection performance and Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index
Measure (SSIM) for restoration.
Since our algorithm focuses on improving detection performance without sacrificing visual quality, we expect a gain of mAP and as few degradations in PSNR/SSIM as possible.
Baselines: We evaluate our proposed fine-tuning procedure on four restoration models, i.e. , MSBDN [8] and GridDehaze [24] for the dehazing task as well as ZeroDCE [12] and SCI [27] for the low light enhancement task.
We compare our fine-tuning pipeline with the traditional learning with only restoration loss.
To test the generalization of our pipeline, we conduct experiments for two widely used detectors, i.e. , YOLOv3 [31] and Faster-RCNN [32].
We also compare our method against some domain-adaption approaches including DA-YOLO [5], DSNet [15], and IA-YOLO [23].
Implementation Details We use YOLOv3 [47] with Darknet-53 backbone.
We train both
YOLOv3 [31] and Faster-RCNN [32] on the clean images of VOC_fog_train and VOC_dark_train respectively.
Then, their parameters are fixed during fine-tuning image restoration methods.
No data augmentation is utilized in the experiments.
All experiments are conducted on an Nvidia Tesla
V100 PCIe 32G GPU and implemented by PyTorch.
We fine-tuned models with batch 1 at each iteration without cropping since the adversarial attack on detectors requires the whole image as input.
The tuning number of epochs is 10 for all restoration models.
The learning rate is set 1e-4 and the optimizer is ADAM with default settings for all models.
For better learning for restoration, we replace all L1 loss with Charbonnier loss [1], L
[Equation]
(cid:88) i = W (cid:88) j = C (cid:88) k
= (cid:113) (
[Equation]
where λ is a constant for numerical stability.
We set λ as 1e-6.
It is a variant of L1 loss and reported better in restoration [19]. [Figure] [Figure]
4.2 Comparisons 4.2.1 Targeted Attack on Test Set
We first evaluate whether a pseudo ground truth that enable better detection exists by performing adversarial attack on test sets of RTTS, VOC_fog_test, VOC_dark_test, and ExDark.
We compare our method against TOG [6].
The result for YOLOv3 [31] are given in Table 1 and that for FasterRCNN [32] are reported in the supplemental material.
We can observe that both detectors face significant performance degeneration from clean images to corrupted ones.
The mAP of YOLOv decreases by nearly 15% (15.03% and 14.11%) on degraded images compared to clean ones for both
VOC_fog_test and VOC_dark_test.
After the targeted attack with a small perturbation, detection accuracy increases significantly.
For example, the mAP of YOLOv3 increases from 42.77% on RTTS to over 70% after the attack.
Compared to TOG, our algorithm can boost detection performance by 2% - 7% higher.
We show two groups of samples for RTTS in Figure 2 and ExDark in Figure 3.
As shown, the detectors cannot work correctly on corrupted inputs.
But attacked images enable detectors to predict better and more bounding boxes.
Compared to TOG, the images generated by our method lead to more accurate detection predictions.
For example, Faster-RCNN cannot detect any car or bus in
Figure 2d.
However, it can detect 6 cars and 2 buses correctly on the attacked image by our method.
These experiments demonstrate that the theoretical pseudo ground truth that is close to the original ground truth, and enables better detection exists.
4.2.2 Dehazing for Detection
Then we show the experiments of restoration by fine-tuning with pseudo ground truth.
We train restoration methods on VOC_fog_train and VOC_dark_train and test them on VOC_fog_test, RTTS,
VOC_dark_test and ExDark.
During fine-tuning with our proposed pipeline, clean images are replaced by attacked ones.
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] The main results of dehazing for detection by YOLOv3 [31] are shown in Table 2.
We can find that
MSBDN [8] fine-tuned by our algorithm can boost detection performance compared to conventional training methods with original ground truth and fine-tuning with TOG attack.
Similar tendencies exist for GridDehaze [24].
We show two groups of examples in Figure 2.
Our method yields better detection accuracy and keeps adequate restoration performance.
However, previous work [23] as shown in Figure 4b outputs unnatural images with many artifacts and noises. [Figure] [Table] [Figure]
The last four columns perform detection by YOLOv3[31].
4.2.3 Low Light Enhancement for Detection
We show the main results of low light enhancement for object detection by YOLOv3 [31] in Table 3.
Compared to conventional training with original ground truth and fine-tuning with TOG attacked,
SCI [27] fine-tuned by our method boosts detection performance.
Similar tendencies exist for
ZeroDCE [12].
We show two groups of examples in Figure 3.
Our method yields better detection accuracy and, at the same time, keeps fair restoration performance.
4.2.4 Limitation
[Figure]
We note that directly performing the adversarial attack on test sets leads to a significant boost in detection results as shown in Table 1.
However, the restoration models learning the map from corrupted image to pseudo ground truth cannot yield such a significant improvement.
The reason may be the restoration models are incapable of learning perfectly the pattern of attacked pseudo ground truth.
The error between original and pseudo ground truths, which makes the attack ineffective.
We show a failure case in Figure 6.
Though the image is less hazy after restoration, the detector cannot obtain a better prediction on the recovered image.
Conclusion
In this work we present a training procedure for image restoration aimed at better performance for downstream detectors without much visual quality degeneration.
The training is supervised by pseudo ground truth generated by the proposed ADAM-like targeted adversarial attack.
We conduct extensive experiments in two restoration tasks, i e. , dehazing and low light enhancement for both oneand two-stage object detectors.
We show our training pipeline effectively improves the detection performance gain of restoration algorithms as a pre-processing step compared to conventional training and other existing works.
Acknowledgments and Disclosure of Funding
Supported by the National Key R&D Program of China under Grant 2018AAA0102503, National
Natural Science Foundation of China (No. 62025604, 62172409, U1803264), Beijing Natural Science
Foundation (No. M22006).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We have accurately expressed the main idea of the work.
(b) Did you describe the limitations of your work?
[Yes] It is described in Section 4.2.4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work does not have any potential negative society impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We include the assumptions in Section 3.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] We include complete proofs in Section 3.1 and supplemental materials.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the code in supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We include all the training details in Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In Section 4.1. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]