Paper:
We propose a novel defense against all existing gradient based adversarial attacks on deep neural networks for image classification problems. Our defense is based on a combination of deep neural networks and simple image transformations. While straight forward in implementation, this defense yields a unique security property which we term buffer zones. In this paper, we formalize the concept of buffer zones. We argue that our defense based on buffer zones is secure against state-of-the-art black box attacks. We are able to achieve this security even when the adversary has access to the entire original training data set and unlimited query access to the defense. We verify our security claims through experimentation using FashionMNIST, CIFAR-10 and CIFAR-100. We demonstrate < 10% attack success rate -significantly lower than what other well-known defenses offer -at only a price of a 15-20% drop in clean accuracy. By using a new intuitive metric we explain why this trade-off offers a significant improvement over prior work.

Review:
This paper proposes a defense against black box adversarial attack. The authors train an ensemble of deep networks, and output a null label when the ensemble disagree. Success of adversarial attack is defined as the ensemble outputs an incorrect label that is not null. The authors experimentally show improved robustness to adversarial attack. 
The idea is itself new, but very similar ideas are well known in the literature, and it is difficult to conclude that the proposed approach is superior. Several examples are:
Defense by majority vote with ensembles has appeared several times in the literature (e.g. Pang et al 2019). The pro is that this paper proposes a novel way to create an ensemble by applying random linear transformations and rescaling to the input. But it is not clear this is superior compared to existing methods. 
Randomized smoothing (Cohen et al, 2019) guarantees smoothness of the classifier (and thus robustness to perturbation attack under certain norms). Note that randomized smoothing  provides certified guarantee against a stronger attack model (white box); it also guarantees the size of the margin (or buffer as the authors call it). The intuition of this paper is based on similar ideas so it seems necessary to at least compare with randomized smoothing. 
Outputting a null label is a major workhorse of adversarial defense. For example, previous work use a generative model to detect out of distribution samples; use a calibrated classifier to output null when low confidence. 
Because the idea is only mildly interesting, good experimental support becomes crucial. However, I think there are several short-comings with the experiments:
The experiment contains only one (fairly old) attack method. Several recent alternatives such as SimBA (Guo et al 2019) can make the experiments more convincing. 
The architecture is no longer the same for the target model (which is an ensemble with an additional random transformations) and surrogate model. It is unclear if the improvement is simply because of the difference in architecture. 
The comparison to baselines seem unfair because it seems that the compared baselines do not have the option of outputting the null label. For example, a simple baseline of randomized smoothing + output the null label if the logit scores are below a threshold can make the story much stronger.
	
Minor comments:
Several suggestions on writing: the introduction contains much technical detail and even experimental results, and these are repeated again in later sections. The experimental section has many minor implementation details that could go into the appendix.