Paper:
COST-SENSITIVE ROBUSTNESS AGAINST ADVERSARIAL EXAMPLES
Several recent works have developed methods for training classifiers that are certifiably robust against norm-bounded adversarial perturbations. These methods assume that all the adversarial transformations are equally important, which is seldom the case in real-world applications. We advocate for cost-sensitive robustness as the criteria for measuring the classifier's performance for tasks where some adversarial transformation are more important than others. We encode the potential harm of each adversarial transformation in a cost matrix, and propose a general objective function to adapt the robust training method of  to optimize for cost-sensitive robustness. Our experiments on simple MNIST and CIFAR10 models with a variety of cost matrices show that the proposed approach can produce models with substantially reduced cost-sensitive robust error, while maintaining classification accuracy.

Review:
** review score incremented following discussion below **
Strengths:
Well written and clear paper
Intuition is strong: not all source-target class pairs are as beneficial to find adversarial examples for 
Weaknesses:
Cost matrices choices feel a bit arbitrary in experiments
CIFAR experiments still use very small norm-balls
The submission builds on seminal work by Dalvi et al. (2004), which studied cost-sensitive adversaries in the context of spam detection. In particular, it extends the approach to certifiable robustness introduced by Wong and Kolter with a cost matrix that specifies for each pair of source-target classes whether the model should be robust to adversarial examples that are able to take an input from the source class to the target (or conversely whether these adversarial examples are of interest to an adversary).
While the presentation of the paper is overall of great quality, some elements from the certified robustness literature could be reminded in order to ensure that the paper is self-contained. For instance, it is unclear how the guaranteed lower bound is derived without reading prior work. Adding this information in the present submission would make it easier for the reader to follow not only Sections 3.1 and 3.2 but also the computations behind Figure 1.b. 
The experiments results are clearly presented but some of the details of the experimental setup are not always justified. If you are able to clarify the following choices in your rebuttal, this would help revise my review. First, the choice of cost matrices feels a bit arbitrary and somewhat cyclical. For instance, binary cost matrices for MNIST are chosen according to results found in Figure 1.b, but then later the same bounds are used to evaluate the performance of the approach. Yet, adversarial incentives may not be directly correlated with the “hardness” of a source-target class pair as measured in Figure 1.b. The real-valued cost matrices are better justified in that respect. Second, would you be able to provide additional justification or analysis of the choice of the epsilon parameter for CIFAR-10? For MNIST, you were able to improve the epsilon parameter from epsilon=0.1 to epsilon=0.2 but for CIFAR-10 the epsilon parameter is identical to Wong et al. Does that indicate that the results presented in this paper do not scale beyond simple datasets like MNIST?
Minor comments:
P2: The definition of adversarial examples given in Section 2.2 is a bit too restrictive, and in particular only applies to the vision domain. Adversarial examples are usually described as any test input manipulated by an adversary to force a model to mispredict.
P3: typo in “optimzation” 
P5: trade off -> trade-off 
P8: the font used in Figure 2 is small and hard to read when printed.