Certified defenses based on convex relaxations are an established technique for training provably robust models. The key component is the choice of relaxation, varying from simple intervals to tight polyhedra. Paradoxically, however, it was empirically observed that training with tighter relaxations can worsen certified robustness. While several methods were designed to partially mitigate this issue, the underlying causes are poorly understood. In this work we investigate the above phenomenon and show that tightness  may not be the determining factor for reduced certified robustness . Concretely, we identify two key features of relaxations that impact training dynamics: continuity and sensitivity. We then experimentally demonstrate that these two factors  explain the drop in certified robustness when using popular relaxations. Further, we show, for the first time, that it is possible to successfully train with tighter relaxations (i.e., triangle), a result supported by our two properties. Overall, we believe the insights of this work can help drive the systematic discovery of new effective certified defenses .
Certified defenses based on convex relaxations are an established technique for training provably robust models. The key component is the choice of relaxation, varying from simple intervals to tight polyhedra. Paradoxically, however, training with tighter relaxations can often lead to worse certified robustness. The poor understanding of this paradox has forced recent state-of-the-art certified defenses to focus on designing various heuristics in order to mitigate its effects. In contrast, in this paper we study the underlying causes and show that tightness alone may not be the determining factor . Concretely, we identify two key properties of relaxations that impact training dynamics: continuity and sensitivity. Our extensive experimental evaluation demonstrates that these two factors , observed alongside tightness, explain the drop in certified robustness for popular relaxations. Further, we investigate the possibility of designing and training with relaxations that are tight, continuous and not sensitive. We believe the insights of this work can help drive the principled discovery of new and effective certified defense mechanisms .
D
it was empirically observed that
R
worsen
often lead to worse
R
While several methods were designed to partially mitigate this issue, the underlying causes are poorly understood. In this work we investigate the above phenomenon
The poor understanding of this paradox has forced recent state-of-the-art certified defenses to focus on designing various heuristics in order to mitigate its effects. In contrast, in this paper we study the underlying causes
A
alone
D
for reduced certified robustness
R
features
properties
R
We then experimentally demonstrate
Our extensive experimental evaluation demonstrates
A
, observed alongside tightness,
R
when using
for
R
show, for the first time, that it is possible to successfully train with tighter relaxations (i.e., triangle), a result supported by our two properties. Overall, we
investigate the possibility of designing and training with relaxations that are tight, continuous and not sensitive. We
R
systematic
principled
R
effective certified defenses
and effective certified defense mechanisms