Paper:
We study the most practical problem setup for evaluating adversarial robustness of a machine learning system with limited access: the hard-label black-box attack setting for generating adversarial examples, where limited model queries are allowed and only the decision is provided to a queried data input. Several algorithms have been proposed for this problem but they typically require huge amount (>20,000) of queries for attacking one example. Among them, one of the state-of-the-art approaches  showed that hard-label attack can be modeled as an optimization problem where the objective function can be evaluated by binary search with additional model queries, thereby a zeroth order optimization algorithm can be applied. In this paper, we adopt the same optimization formulation but propose to directly estimate the sign of gradient at any direction instead of the gradient itself, which enjoys the benefit of single query. Using this single query oracle for retrieving sign of directional derivative, we develop a novel query-efficient Sign-OPT approach for hard-label black-box attack. We provide a convergence analysis of the new algorithm and conduct experiments on several models on MNIST, CIFAR-10 and ImageNet. We find that Sign-OPT attack consistently requires 5× to 10× fewer queries when compared to the current state-of-the-art approaches, and usually converges to an adversarial example with smaller perturbation.

Review:
In this paper, the authors propose a new algorithm for evaluating adversarial robustness of black-box models. The aim of the proposed algorithm, SIGN-OPT is generating adversarial examples as close as possible to the decision boundary using as less queries of the black-box model as possible.
The authors follow the approach of (Cheng et al 2019) by modeling the problem as an optimization problem, where the objective function is to find the direction with the shortest distance to the decision boundary. They propose a smart modification of the previous approach by evaluating the sign of the gradient rather than the gradient itself. The advantage is that the sign of the gradient can be evaluated using a single query while the estimation of the gradient needs many queries. 
The authors have analyzed the proposed algorithm. They showed that using SIGN-OPT, the expectation of the gradient tends to zero in $O(1/\sqrt(T))$, meaning that a (local) minimum is reached.
The algorithm is favorably compared with the state-of-the-art on three image test sets (MNIST, CIFAR-10n ImageNet).
This paper is technically sound, well-written and propose an interesting modification of a previous algorithm. I vote for acceptance.
However, I have some concerns:
-	I think that the L-smoothness assumption should be discussed. Is it realistic for Deep Learning models? Does it hold for the three attacked CNN networks?
-	The analytical results of the previous algorithm RGF and the proposed algorithm SIGN-OPT are not compared and discussed. It is a pity.