  Nowadays, general object detectors like YOLO and Faster R-CNN as well as
their variants are widely exploited in many applications. Many works have
revealed that these detectors are extremely vulnerable to adversarial patch
attacks. The perturbed regions generated by previous patch-based attack works
on object detectors are very large which are not necessary for attacking and
perceptible for human eyes. To generate much less but more efficient
perturbation, we propose a novel patch-based method for attacking general
object detectors. Firstly, we propose a patch selection and refining scheme to
find the pixels which have the greatest importance for attack and remove the
inconsequential perturbations gradually. Then, for a stable ensemble attack, we
balance the gradients of detectors to avoid over-optimizing one of them during
the training phase. Our RPAttack can achieve an amazing missed detection rate
of 100% for both Yolo v4 and Faster R-CNN while only modifies 0.32% pixels on
VOC 2007 test set. Our code is available at
https://github.com/VDIGPKU/RPAttack.
