Rethinking Image Restoration for Object Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Although image restoration has achieved significant progress, its potential toassist object detectors in adverse imaging conditions lacks enough attention inresearch community.
It is reported that the existing image restoration methodscannot improve the object detector performance and sometimes even reduce thedetection performance.
To address the issue, we propose a targeted adversarialattack in the restoration procedure to boost object detection performance afterrestoration.
Specifically, we present an ADAM-like adversarial attack to generatepseudo ground truth for restoration fine-tuning.
Resultant restored images areclose to original sharp images and at the same time, lead to better results of objectdetection.
We conduct extensive experiments in image dehazing and low lightenhancement and show the superiority of our method over conventional trainingand other domain adaptation and multi-task methods.
The proposed pipeline canbe applied to all restoration methods and both one- and two-stage detectors
Introduction
Image quality degradation is common in outdoor captured images and many image restorationmethods have been developed to automatically recover sharp details from the degraded images.
Typically, the goal of image restoration is to generate sharper results from which human visioncan better identify the structural details and objects in the images.
However, in many applications,large-scale images are collected for high-level computer vision tasks ( e.g. , object detection, semanticsegmentation, and autonomous driving) other than a visual examination by human.
Though significantprogress has been made in terms of the visual quality, its ability to serve as a pre-processing step fordownstream vision tasks lacks enough research attention.
Object detection has achieved remarkable success as the boosting of deep neural networks.
Manysuccessful models, e.g. , Faster-RCNN [32] and YOLOv3 [31], have been developed and deployedin real-world applications such as autonomous driving [4, 38].
However, in the settings of thecommonly used benchmark for object detection, poor weather and imaging conditions are intentionally avoided [7] or insufficient due to data bias [16, 9].
The unbalanced data distribution towardclean images leads to a domain shift between properly captured images and low-quality ones inadverse environments [36].
The performance of object detector trained on high-quality samples thusdowngrades significantly in adverse imaging and weather conditions such as haze, rain, and low lightexposure [23, 5] since the weather-related deterioration features distort captured image [29, 47].
Many existing solutions to the issue lie in domain adaptation from clean images (source domain)to corrupted ones (target domain) [5, 13, 36].
They assume there exists a distribution shift betweenclean images and those in poor conditions, and adapt detectors by unsupervised weather-specificpriors or domain adaptation components to align the features of two distributions to reduce domain discrepancy.
However, the detector after adaptation is found to perform worse in the source domainthan that without adaptation [23] since adapted detectors may focus on weather-relevant deteriorationinformation.
Another track to tackle the issue is to jointly train a simple visual quality enhancementmodule together with detection network [5, 23].
However, these modules cannot provide satisfactoryresults compared to powerful restoration methods.
Many artifacts and noise patterns occur whenapplying the enhancement module.
It is also challenging to balance visual quality and detectionperformance by tuning hyper-parameters during training.
Since the detector is trained on corruptedsamples or the combination of corrupted and clean samples, its performance on clean samples maydecrease, similar to the case of domain adaptation.
Different from previous works modifying object detectors, we address the issue by adapting restorationalgorithms such that recovered images can achieve both good visual quality and better detection resultsfor detectors.
Though image restoration methods produce satisfying results from the perspectiveof human eyes, detectors may consider them as variables following the distribution of an artificialneural network rather than the real-world, thus being confused by such a covariate shift.
Therefore,finding a pseudo ground truth close to the original sharp image potentially improves detection.
Tofind such pseudo ground truth, a targeted adversarial attack can be employed to add an invisiblysmall perturbation to the original ground truth, on which the detector will give our desired prediction.
The resultant adversarial examples are obtained by minimizing the loss of object detectors overperturbation, and can serve as pseudo ground truth for restoration algorithms.
Such perturbationis trivial, so restoration models are not impacted to the greatest extent.
During the fine-tuningstage, only the weights of restoration methods are updated and detectors keep unchanged.
Thus theperformance of detectors will not decline in the clean domain.
Note that the goal of this paper isnot the development of a new image restoration algorithm or a new object detection model.
Instead,we propose a targeted attack preprocessing of image ground truth to help improve the accuracyof downstream object detection.
Specifically, we propose a momentum-based Adam [17]-variantadversarial example generation method to estimate pseudo ground truth.
Our contributions can be summarized as following:
We propose a
fine-tuning pipeline to improve detection performance and retain visuallysatisfactory restoration results.
Specifically, we show multi-task optimization problem ofbalancing both detection and restoration can be decomposed into a targeted adversarial attackgeneration and a restoration problem.
The pseudo ground truth generated by adversarialexample generation is used for fine-tuning.2.
We design a momentum-based ADAM-like iterative targeted adversarial example generationalgorithm to estimate pseudo ground truth for restoration models fine-tuning.
Rather thanupdating samples by first-order gradient in existing work, we compute ADAM-variantclipped and signed gradient with momentum estimation.3.
We conduct extensive experiments in two image restoration tasks, i.e. , dehazing and lowlight enhancement for both one- and two-stage object detectors.
We first evaluate theeffect of our adversarial attack algorithm compared to existing ones on test sets.
We thenshow our proposed fine-tuning pipeline effectively improves the detection performance gainof restoration algorithms as a pre-processing step of detectors compared to conventionaltraining and other existing works.
Related Works
2.1 Restoration for Detection
In the image restoration task, many works have achieved promising performance in terms of visualquality and quantitative evaluations [24, 8, 20, 35, 33].
Though many of them mention their potentialusage as a pre-processing for downstream high-level tasks ( e.g. , object detection and segmentation),only a few of them experiments on how much gain of detection accuracy can be acquired before andafter the dehazing step.
Dong et al.
[8] simulate haze on KITTI [10] and show their method removesthe synthetic haze effectively and can improve YOLOv3 [31] detection predictions.
Li et al.
[20] publicize the RESIDE dataset, which contains a subset in real-world hazy environment for evaluatingobject detection on dehazed results.
A recent work [23] appends a differentiable image processingmodule consisting of multiple traditional filters on YOLOv3, and trains the module and YOLOv3together on the combination of clean and hazy samples.
Due to the limitation of filters, it generatesmany noises and artifacts and thus returns many false positive detection predictions.
In low-light environments, several object detection and semantic segmentation datasets have beencollected.
Yang et al.
[46] create a benchmark for face detection in real-world low light condition.
Loh et al.
[26] publish a object detection benchmark in low light environment.
Recently, Sakaridiset al.
[34] collect a comprehensive semantic segmentation dataset in dark imaging condition.
Totackle low-light enhancement task, ZeroDCE [12] is developed by formulating light enhancementas a task of image-specific curve estimation in the pixel-wise range.
Its performance of helpingface detection in low-light conditions is tested on DarkFace [46].
Ma et al.
[27] recently develop anew Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brighteningimages in real-world low-light scenarios.
They show its improvement for low-light face detectionand segmentation tasks on ACDC [34] and DarkFace [46] datasets.
The above-mentioned works either merely experiment on synthetic datasets or have to tune detectionmodels together with a restoration module, which limits the generalizability and extensibility oftheir works on real-world benchmarks and various tasks.
In this work, we focus on proposing aplug-and-play targeted attack module for restoration algorithms to improve the performance of objectdetection without tuning detection models.
2.2 Adversarial Attack on Detector
After Szegedy et al.
[37] finds that adding an imperceptible perturbation to an image may fool neuralnetwork classifier, many adversarial attack generation methods are presented in classification [11, 30,28, 3, 18].
Fewer works about adversarial attack on object detector are proposed [25] because of itsmore complicated architecture and loss terms [41, 2].
Some works succeed in fooling detectors byeither wearable patch [40, 45, 42] or non-planar patch [50, 48, 43, 14, 39].
Only limited works areabout targeted attack.
Dense Adversary Generation (DAG)
[44], Robust Adversarial Perturbation(RAP)
[22], and Contextual Adversarial Perturbation (CAP) [49] are designed for the two-stagedetectors consisting of region proposal network.
Other works such as Unified and Efficient Adversary(UEA)
[41] and Targeted adversarial Objectness Gradient attack (TOG)
[6] can attack both one-stageand two-stage detectors with more generic settings.
UEA is a Generative Adversarial Networkframework trained with class loss and feature loss that generates transferable adversarial imagesand videos.
TOG [6] consists of three relatively powerful adversarial attack algorithms for objectvanishing, object-fabrication, and object-mislabeling attacks, respectively.
A recent work [2] developsa content-aware blackbox adversarial attack framework based on TOG.
In our work, we use TOGas the baseline for whitebox targeted adversarial example generation due to its flexibility andeffectiveness in conducting extensive experiments.
We assume detectors are whitebox because ourgoal is to improve its performance rather than fool it.
Proposed Method
3.1 Pseudo Ground Truth Generation for Restoration
Denote a restoration model with parameters ω as R and a detector with parameter θ as D .
Suppose adataset for training contains three components: corrupted images x , corresponding clean images ˆx ,and detection annotations ˆy .
Given that D only learns on clean samples { ˆx , ˆy } , the prediction of thedetector on corrupted samples D ( x ) is probably incorrect.
Our aim is to fine-tune a restorer R suchthat the recovered image is close to clean image, i.e. , R ( x ) ≈ ˆx , and also the detector give correctprediction on the recovered image, i.e. , D [ R ( x )]
= ˆy .
Traditional supervised restoration problem is not involved with the detection objective and can begenerally formulated as:
[Equation]
[Figure]
However, solely training with the objective 1 cannot guarantee a satisfactory result in subsequentobject detection since the covariate shift and artifacts as reported in [20, 23, 21].
Given that restorationis an ill-posed problem, various possible recovered images exist for a degraded one.
Considering theinevitable image noises due to hardware limitations during the image acquisition and transmissionstage especially in adverse weather and imaging conditions, captured clean images can unnecessarilybe regarded as a perfect ground truth.
Unlike conventional restoration methods, our goal is to find a balance between visual quality anddetection performance on restored images.
Our objective is to make detector D inference correctly onthe restored image as close as possible to the clean image.
Thus it can be considered as a multi-taskoptimization problem by combining the restoration constraint and object detection loss.
We start byformulating it as follows,
[Equation]
where ˜x = R ( x ) is the recovered image, s is a slackness coefficient indicating the allowed distancefrom the ground truth ˆx .
The constraint optimization problem cannot be solved easily due to theconstraint.
Thus we formulate its dual function as below by introducing a dual variable α ≥ 0 as,
[Equation]
Theorem 1 For any s > 0 , there exists α ≥ 0 such that the optimal solution of (2), ˜x ∗ , is alsooptimal for (3) , and vise versa.
The proof of Theorem 1 is attached in the supplemental materials.
The main idea is to use Lagrangianand a strong duality assumption.
By Theorem 1, we can say the original problem Eq.2 and the dualproblem is equivalent.
However, the multi-task training faces difficulty in tuning the hyper-parameterα , and an improper α leads to either poor detection or restoration results.
Figure 1 shows a casewhere optimizing d 1 cannot ensure correct detection.
Thus we further reformulate the objective and bypass the issue.
By substituting β = 1 α , we get
[Equation]
Output
Assign β 1 = 0 .
9 , β 2 = 0 .
999 , ϵ = 1 e −
Initialize m 0 = 0 , v 0 = 0 , z = rand ( − δ, δ ) ; x ← clip ( x + z , 0 , 1) foreach ( x , ˆy ) in U do foreach t in 1 ...
T do
[Equation]
[Equation]
t
[Equation]
) ˆv ← v t / (1 − β t 2 )
By Theorem 1 again, we have
[Equation] [Equation]
where r is a small slackness coefficient that regularizes the detection performance.
By introducing anintermediate variable ˆx ∗ that is close enough to ˆx , we have the expression of optimization,min ˜x ∥ ˆx ∗ − ˜x ∥ , where ˆx ∗ =
arg min x
[Equation]
[Equation]
The solutions of (5) and (6) are equivalent when ˜x ∗ =
ˆx ∗ .
The problem can thus be separated intotwo optimization steps,
[Equation]
Therefore, (7) is an objective of targeted adversarial attack if we substitute for x = ˆx + z and addbox constraint ∥ x ∥ ≤ 1 .
Then, (7) can yield a detector-friendly sample close to the original groundtruth, and (8) forces restoration models to learn the pattern mapping corrupted samples to the pseudoground truth.
The solution of (5) can be approximately computed by the fine-tuning pipeline ofsequentially optimizing (7) and (8).
3.2 Targeted Adversarial Attack on Object Detector
Object detector D receives an input image x and predicts label with n detected objects, i.e. , y ={ ( b x , b y , b H , b W , C, p ) i | i
= 1 ...n } , where ( b x , b y , b H , b W ) represents the location and region ofbounding box for i -th object, C is classification result and p is prediction confidence.
The existingmethod, TOG [6] designs a group of loss terms specific for object detector.
The adversarial exampleis updated iteratively by signed and clipped first-order gradient of the loss term as,
[Equation]
9) where λ is the step size of perturbation update and δ is the allowed perturbation scale centeredat original input x .
However, we note that the first-order gradient descent used in TOG may getstuck at sub-optimal or converge slowly.
Therefore, we introduce the momentum-based optimizer
ADAM [17] to optimize adversarial examples during updating perturbation, which is widely used in optimizing parameters of neural networks during training.
It can adaptively adjust learning rateby second-order momentum and overcome sub-optimal with cumulative momentum.
Therefore, wereformulate (9) into
[Equation]
Finally, the generated pseudo ground truth ˆx ∗ are used to fine-tune restoration models.
The mainstep of the targeted adversarial attack is presented in Algorithm 1.
The convergence analysis of thealgorithm is discussed in the supplemental material.
Experiments
4.1 Experimental Settings
We conduct experiments on two image restoration tasks, i.e. , image dehazing and low-light imageenhancement, and two detectors of YOLOv3 [31] and Faster-RCNN [32].
Dehazing Dataset: RTTS [20] is a real-world hazy dataset with detection annotation for test purpose.
The object labels contain five categories, i.e. , person, car, bus, bicycle, and motorcycle .
To construct atraining set with both ground truth clean image and bounding box annotation, we select those PASCAL
VOC images with the above-mentioned five classes and simulate haze on them following [23].
Weonly simulate two scales of haze rather than ten in [23], and thus our training set is only 20% of theirs.
RTTS contains 4,322 real-world hazy images.
VOC_fog_train and VOC_fog_test consist of 16,222and 2,734 synthetic hazy images respectively.
Low-light Enhancement Dataset: ExDark [26] is a natural low light dataset for object detectionconsisting of ten classes, i.e. , bicycle, boat, bottle, bus, car, cat, chair, dog, motorbike, person .
Similarto dehazing task, we select those samples in PASCAL VOC with these ten categories and simulatelow light corruption to form the training set, dubbed as VOC_dark_train.
ExDark contains 2,563real-world dark images.
VOC_dark_train and VOC_dark_test consist of 12,334 and 3,760 syntheticdark images respectively.
Metrics: We adopt mean average precision (mAP) with an IOU threshold 50% as a metric fordetection performance and Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index
Measure (SSIM) for restoration.
Since our algorithm focuses on improving detection performancewithout sacrificing visual quality, we expect a gain of mAP and as few degradations in PSNR/SSIMas possible.
Baselines: We evaluate our proposed fine-tuning procedure on four restoration models, i.e. , MSBDN [8] and GridDehaze [24] for the dehazing task as well as ZeroDCE [12] and SCI [27] for the lowlight enhancement task.
We compare our fine-tuning pipeline with the traditional learning with onlyrestoration loss.
To test the generalization of our pipeline, we conduct experiments for two widelyused detectors, i.e. , YOLOv3 [31] and Faster-RCNN [32].
We also compare our method against somedomain-adaption approaches including DA-YOLO [5], DSNet [15], and IA-YOLO [23].
Implementation Details We use YOLOv3 [47] with Darknet-53 backbone.
We train both
YOLOv3 [31] and Faster-RCNN [32] on the clean images of VOC_fog_train and VOC_dark_trainrespectively.
Then, their parameters are fixed during fine-tuning image restoration methods.
Nodata augmentation is utilized in the experiments.
All experiments are conducted on an Nvidia Tesla
V100 PCIe 32G GPU and implemented by PyTorch.
We fine-tuned models with batch 1 at eachiteration without cropping since the adversarial attack on detectors requires the whole image as input.
The tuning number of epochs is 10 for all restoration models.
The learning rate is set 1e-4 and theoptimizer is ADAM with default settings for all models.
For better learning for restoration, we replaceall L1 loss with Charbonnier loss [1],
[Equation]
[Figure] [Figure]
where λ is a constant for numerical stability.
We set λ as 1e-6.
It is a variant of L1 loss and reportedbetter in restoration [19].
4.2 Comparisons
4.2.1 Targeted Attack on Test Set
We first evaluate whether a pseudo ground truth that enable better detection exists by performingadversarial attack on test sets of RTTS, VOC_fog_test, VOC_dark_test, and ExDark.
We compareour method against TOG [6].
The result for YOLOv3 [31] are given in Table 1 and that for FasterRCNN [32] are reported in the supplemental material.
We can observe that both detectors facesignificant performance degeneration from clean images to corrupted ones.
The mAP of YOLOv3decreases by nearly 15% (15.03% and 14.11%) for both VOC_fog_test and VOC_dark_test.
Afterthe targeted attack with a small perturbation, detection accuracy increases significantly.
For example,the mAP of YOLOv3 increases from 42.77% on RTTS to over 70% after the attack.
Compared to TOG, our algorithm can boost detection performance by 2% - 7% higher.
We show two groups of samples for RTTS in Figure 2 and ExDark in Figure 3.
As shown, thedetectors cannot work correctly on corrupted inputs.
But attacked images enable detectors to predictbetter and more bounding boxes.
Compared to TOG, the images generated by our method lead tomore accurate detection predictions.
For example, Faster-RCNN cannot detect any car or bus in
Figure 2d.
However, it can detect 6 cars and 2 buses correctly on the attacked image by our method.
These experiments demonstrate that the theoretical pseudo ground truth that is close to the originalground truth, and enables better detection exists.
4.2.2 Dehazing for Detection
Then we show the experiments of restoration by fine-tuning with pseudo ground truth.
We trainrestoration methods on VOC_fog_train and VOC_dark_train and test them on VOC_fog_test, RTTS, [Table]
[Table]
VOC_dark_test and ExDark.
During fine-tuning with our proposed pipeline, clean images arereplaced by attacked ones.
The main results of dehazing for detection by YOLOv3 [31] are shown in Table 2.
We can find that
MSBDN [8] fine-tuned by our algorithm can boost detection performance compared to conventionaltraining methods with original ground truth and fine-tuning with TOG attack.
Similar tendenciesexist for GridDehaze [24].
We show two groups of examples in Figure 2.
Our method yields betterdetection accuracy and keeps adequate restoration performance.
However, previous work [23] asshown in Figure 4b outputs unnatural images with many artifacts and noises.
4.2.3 Low Light Enhancement for Detection
We show the main results of low light enhancement for object detection by YOLOv3 [31] in Table 3.
Compared to conventional training with original ground truth and fine-tuning with TOG attacked,
SCI [27] fine-tuned by our method boosts detection performance.
Similar tendencies exist for
[Figure]
[Table]
[Figure]
[Figure]
The last four columns perform detection by YOLOv3[31].
ZeroDCE [12].
We show two groups of examples in Figure 3.
Our method yields better detectionaccuracy and, at the same time, keeps fair restoration performance.
4.2.4 Limitation
[Figure]
[8] with our fine-tuning pipeline.
We note that directly performing the adversarial attack on test sets leads to a significant boost indetection results as shown in Table 1.
However, the restoration models learning the map fromcorrupted image to pseudo ground truth cannot yield such a significant improvement.
The reason maybe the restoration models are incapable of learning perfectly the pattern of attacked pseudo groundtruth.
The error between original and pseudo ground truths, which makes the attack ineffective.
Weshow a failure case in Figure 6.
Though the image is less hazy after restoration, the detector cannotobtain a better prediction on the recovered image.
Conclusion
In this work, we present a training procedure for image restoration aimed at better performancefor downstream detectors without much visual quality degeneration.
The training is supervised bypseudo ground truth generated by the proposed ADAM-like targeted adversarial attack.
We conductextensive experiments in two restoration tasks, i.e. , dehazing and low light enhancement for both oneand two-stage object detectors.
We show our training pipeline effectively improves the detectionperformance gain of restoration algorithms as a pre-processing step compared to conventional trainingand other existing works.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We have accurately expressed the main idea of thework.(b)
Did you describe the limitations of your work?
[Yes] We have described it in Section 4.2.4.(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workdoes not have any potential negative society impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We include theassumptions in Section 3.1.(b)
Did you include complete proofs of all theoretical results?
[Yes] We include completeproofs in Section 3.1 and supplemental materials.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include the
code in supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We include all the training details in Section 4.1.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In Section 4.1.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
[Table]
[Table] [Table]