Paper: Amicable Perturbations (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the paper seems limited. While the concept of amicable perturbations is interesting, it appears to be a minor extension of existing techniques for crafting actionable counterfactual explanations.

Author response:
Thank you for your input.  We hope that our comment titled  "Clarification Regarding Novelty of Our Framework" can help explain how our set up differs significantly from previous works on counterfactuals especially in our focus on changing true class probability's instead of merely effecting the classifier's decision (as is the case with most works regarding counterfactual explanations).

Reviewer comment (Evaluation):
The method's robustness against potential countermeasures is not discussed. What if adversarial training is conducted on the classifier to improve the robustness against adversarial examples? Can the proposed amicable perturbations still work?

Author response:
With regards to your comment regarding robustness against counter measures: in our set up it is assumed that the entity crafting amicable perturbations is also the creator of the classifier and there is no adversary to deploy countermeasures.  Instead we are focus on ensuring the creator of an amicable perturbation doesn't inadvertently act as an adversary.  If the classifier was made more robust against adversarial attacks, this would increase our confidence in amicable perturbations. Unfortunately the task of creating robust classifiers is very difficult in general and the topic of robust classifiers for tabular data sets specifically is still in its infancy.  We instead seek to leverage our knowledge of both the original ($\mathbf{x}$) and modified ($\tilde{\mathbf{x}}$) data points to device a defence that is not available in the regular adversarial setting.

Reviewer comment (Theory):
Is there any theoretical analysis that can explain why the proposed verification procedure can ensure that amicable perturbations produce truly different outcomes?

Author response:
**Question**: Is there any theoretical analysis that can explain why the proposed verification procedure can ensure that amicable perturbations produce truly different outcomes?

**Answer**: We elaborate on the theory behind the verification method in the comment titled "Clarification on the Verifier."  We also have the numerical justification that it is $100\%$ effective at identifying the adversarial attacks we created for our data set.

Reviewer comment (Theory):
The primary weakness is the lack of distinction between correlation and causation. The idea of altering an instance $x$ to $\tilde x$ does not really mean that $p(y|\tilde x)$ can be estimated based on the original joint distribution $p(y, x)$. In fact, this is precisely the type of questions the literature in causal analysis focuses on. Amicable perturbation corresponds to what is commonly referred to as do-queries; see for instance Pearl's introduction to causal analysis (https://pubmed.ncbi.nlm.nih.gov/20305706/). The main takeaway is that $p(y|\tilde x)$ (post-intervention) is not necessarily the same distribution as pre-intervention.

Author response:
Thank you for your input.  We appreciate that causality is an important concern in the field of counterfactuals which is gaining more attention.  We believe, however, that the concern of adversarially vulnerable classifiers is also important and worth exploration (as is the goal of this paper).  Although a classifier is accurate in  making predictions on naturally occurring data ($M(\mathbf{x})\approx\mathbb{P}_\mathcal{D}(\mathbf{y}|\mathbf{x})$) it might be inaccurate in making classification on artificially modified data ($M(\tilde{\mathbf{x}}) \not\approx \mathbb{P}_D(\tilde{\mathbf{y}}|\tilde{\mathbf{x}})$) even if $\mathcal{X}$ was created only by adjusting causal features (perhaps based off of a structural causal model) because of weaknesses in the classifier.  More details are found in our comment titled "Clarification Regarding Trustworthiness."  We hope that work in the future will incorporate both the idea of causality and the trustworthiness with respect to adversarial vulnerable models we propose in this paper.

Reviewer comment (Theory):
The main argument in the paper is the claim that one can estimate $p(y| do(x))$ based on the original joint distribution $p(y, x)$, which is wrong. This can even be seen in one of the examples the authors mention in Section 4, in which their approach suggests that an individual should move from the Far West to the Great Lakes in order to improve their chances of passing the BAR exam. In the reviewer's opinion, unless this is addressed, the motivation behind the work is questionable.

Author response:
Regarding Causality: While we agree with you that taking causality explicitly into account is desirable, but we believe that this does not undermine the contribution of our framework.  In addition to ensuring causality, there are several other challenging issues when attempting to change the true class probabilities of an input which we address in this work; such as the unreliable nature of classifiers on artificially generated data points (please see our detailed comment on Trustworthiness). We should also mention that our framework is flexible enough to incorporate causality constraints as follows: a) the actionable set defined in the paper can be modified to only allow changes to causally related features; b) the verification procedure could also be expanded to include a check for causal relationships. We agree that this is an important and interesting point, and we can certainly add this discussion to the revised version of the paper.

We also note, that following the link regarding rates of passing the BAR  (https://lawschooli.com/easiest-bar-exam-to-pass-in-the-us/) leads to an explanation of the results of a law professor's analysis of the relative difficulty of BAR exams after compensating for the different academic backgrounds of students (original analysis here https://witnesseth.typepad.com/blog/2013/04/more-on-the-most-difficult-bar-exams.html).  In this analysis California (Far West) was the single most difficult BAR exam and the states in the Great Lakes area tended to rank among the easier exams (Wisconsin being the second easiest nationwide).  Although it is true that correlated features without a causal link could lead to some inaccurate results, we feel that this concern does not invalidate the contributions of this paper nor the possibility of useful results through our methods,  as illustrated by our method picking up on the same trend found by an independent analysis of law school data.

Reviewer comment (Novelty):
While the authors distance themselves from counterfactual and adversarial examples, their formulation seems to be an extension of those methods. The authors extend those methods to handle the tradeoff between effort and reward. Compare for example Equations 5, 6, and 7. This by itself is not a major limitation. However, it is mentioned here because the narrative of the paper suggests that amicable perturbations are quite different, when they aren't. In Figure 4, for instance, counterfactuals lie *along* the Pareto curve for amicable perturbations.

Author response:
In regards to the novelty of our framework, we acknowledge that our optimization problem (7) is similar to those used by counterfactual and adversarial attack methods (5) and (6) (with a few key differences).  We believe, however, that much of the value and novelty of our framework is found in the clear definition of the problem (Definition 1).  The focus on changing the true probabilities (as a opposed to only focusing on the classifier output) separates us from most works on counterfactuals and all works on adversarial examples.  Although some works on counterfactuals have specified their goal as changing the true class probabilities, we do not believe this problem was well characterized before we introduced the concepts of flexible targets, principled measures of distance to a target and a verification procedure.  More details on this may be found in our comment "Clarification Regarding Framework Novelty."

Reviewer comment (Novelty):
One key novelty in this work is the idea of verification, but this is not well justified. When a model is trained on pairs of examples $(x, x')$ from the same distribution $\mathcal{D}$, there is no clear reason why the model should work well on examples from a different distribution (e.g., $x$ and its perturbation $\tilde x$). If it could work, then the OOD problem would be solved. There is no empirical evidence in the present paper that it works, aside from the fact that it has some effect.

Author response:
Regarding the Verifier:  We appreciate your legitimate point that $\tilde{\textbf{x}}$ may come from a different distribution than the training data, however we hope our detailed comment regarding the classifier explains why we believe our method will be effective at eliminating some ineffective advice.  We fully recognize that data on the results of interventions  (the distribution of $\tilde{\textbf{x}}$) would be very useful.  For the scope of this paper, we did not consider longitudinal data and incorporating  longitudinal data into the verification procedure would be an interesting avenue of future research.  We can certainly include a discussion of this point in the revised paper.

Reviewer comment (Presentation):
Why did the authors present Equations 5 and 6 as separate equations? They seem to be identical except that the notion of distance or loss is instantiated in one case but kept generic in the other.

Author response:
**Question**: Why did the authors present Equations 5 and 6 as separate equations? They seem to be identical except that the notion of distance or loss is instantiated in one case but kept generic in the other.  


**Answer**: The object of showing Equations 5 and 6 individually was to highlight how the methods used for finding counterfactuals (which evolved from Equation 5) and adversarial examples (which evolved from Equation 6) are extremely similar and can even by identical under certain choices for loss functions and norms.  [1] delves deeper into this idea listing exactly which attacks and counterfactual creation methods can be identical or proven to achieve similar outputs.

Reviewer comment (Experiments):
Why does Adult income dataset contain 26000 examples only? It should contain over 48,000 examples.

Author response:
**Question**: Why does Adult income data set contain 26000 examples only? It should contain over 48,000 examples.

**Answer**:  We believe this is a result of our data set being cleaned to remove individuals with missing data.

Reviewer comment (Evaluation):
How is distance $d(x, \tilde x)$ defined in each dataset? For example, the authors claim in Page 8 that one example should be perturbed using "a simple increase in education to the masters level." A masters degree is not a small change.

Author response:
**Question**: How is distance $d_\mathcal{X}(\mathbf{x},\tilde{\mathbf{x}})$ defined in each data set? For example, the authors claim in Page 8 that one example should be perturbed using "a simple increase in education to the masters level." A masters degree is not a small change.
  
**Answer**:  A brief description of $d_\mathcal{X}(\mathbf{x},\tilde{\mathbf{x}})$ in each data set is found on page 7.  The exact function is found on our code.  Based off of reviewer interest we intend on adding more detailed descriptions of the cost functions to the appendix of our revised paper.

  With regards to the specific example listed.  Cost includes the expected number of years to improve ones education ($6$ in this case).  Although this may appear like a large change, it is modest in comparison to the options suggested by a counterfactual method (which suggested over $12$ years of education).  These large changes are simply a result of the original individual being very unlikely to a high income without a significant amount of effort.

Reviewer comment (Theory):
The perturbations generated by the method are not _causal_, unlike previous work [e.g. 1-3]. As such, if the classifier learns some spurious correlations, the proposed method could suggest perturbations that do not causally lead to changes in the true label.

Author response:
Thank you for your input.  We appreciate your concern about causal relations and acknowledge that this is an interesting avenue for future research involving our framework and techniques.  This paper focuses on the separate but equally important concern that modified data points may not produce the anticipated result because ML classifiers are often very inaccurate on modified data point (such as adversarial attacks).  Please see our comment titled "Clarification Regarding Trustworthiness" for more details about this distinction.  This comment also helps explain how our goal is fundamentally different from the goals of [2] and [3].  We also hope the comment titled "Clarification Regarding Trustworthiness" will address some of your concerns regarding the relationship between our paper an previous works.

Reviewer comment (Novelty):
The authors frame their problem of amicable perturbations as an entirely new problem setting. This is an oversell in my opinion, as the proposed setup is essentially the same as a standard algorithmic recourse setup, with the main novelties being that (1) the authors generalize the target set instead of a simple label flip, and (2) the authors propose the second verification step. Note that the idea of adversarially robust recourse has been explored in several prior work [2, 3] which the authors have not referenced.

Author response:
Thank you for your input.  We appreciate your concern about causal relations and acknowledge that this is an interesting avenue for future research involving our framework and techniques.  This paper focuses on the separate but equally important concern that modified data points may not produce the anticipated result because ML classifiers are often very inaccurate on modified data point (such as adversarial attacks).  Please see our comment titled "Clarification Regarding Trustworthiness" for more details about this distinction.  This comment also helps explain how our goal is fundamentally different from the goals of [2] and [3].  We also hope the comment titled "Clarification Regarding Trustworthiness" will address some of your concerns regarding the relationship between our paper an previous works.

Reviewer comment (Experiments):
The authors should conduct an ablation study on e.g. the addition of b(x) and p(x), and the choice of the f-divergence.

Author response:
**Question**: Have the authors considered a projected gradient descent based algorithm, instead of the addition of the $b(\mathbf{x})$ and $p(\mathbf{x})$ penalties?

**Answer**: Yes.  We initially tried projected gradient descent, but we found that if we projected after every step of the gradient descent our output never changed.  If we projected only after the gradient descent had converged, we found that the results were very far from our target (sometimes even further than the original input).  We determined this was because the path of the gradient descent went far outside the actionable set and made highly incoherent inputs (i.e. having multiple values $\gg 1$ in what should be a one-hot encoding).  This often meant that the final projection step erased all progress made my the gradient descent.

We determined it would not be feasible to search for the proper number of steps between projections for each individual optimization task, and instead we adopted the use of $b(\tilde{\mathbf{x}})$ and $p(\tilde{\mathbf{x}})$.

**Question**: For the optimization problem in Eq (7), since the f-divergence is 0 once $M(\tilde{\mathbf{x}})\in T$, it seems like all generated perturbations should be right on the boundary of $T$. However, this is not what happens in practice. Can the authors give some intuition on this?

**Answer**:  That is a very astute observation.  We had initially thought it might be necessary to run the optimization on a slightly smaller target set in order push the amicable perturbations inside the border of $T$.  In practice, however, we found that even though the $\tilde{\mathbf{x}}$ produced immediately after gradient descent lied on the border of $T$, projecting $\tilde{\mathbf{x}}$ onto the coherent space ($cond(\tilde{\mathbf{x}})$) had a tendency to move the amicable perturbation inside of $T$.  It is worth noting that even though the amicable perturbations we produce often lie inside of $T$, we do not produce "overkill" examples that are far inside the border of $T$.  The example counterfactual from the Law School Success data set in page 8 illustrates how other methods do exhibit this "overkill" behavior.

Reviewer comment (Experiments):
The proposed amicable perturbations are mainly suitable for tabular datasets, with well-defined input features. It is not clear how they can be applied to image and text datasets. A discussion on this would be useful.

Author response:
Thank you for your input.  We will look into how are method could be applied to non-tabular data sets as this is an avenue we had not considered.

Reviewer comment (Experiments):
The method supports only differentiable models such as deep neural networks, but cannot be applied to non-differentiable models such gradient-boosted decision trees or random forest. However, the latter models are widely used for tabular data since they often have better performance. The ability to handle such models would provide more flexibility to their proposed method.

Author response:
In regards to your comments on expanding our methods to non-differentiable models, we believe this is a promising avenue for future research.  Many works in counterfactual explanations have sought to find improved methods for solving the mixed integer programming problem (usually involving genetic algorithms or decision  tree specific methods).  We hope that these methods may be adapted to our framework.

Reviewer comment (Reproducibility):
Some of the details in Algorithm 1 are not clear. For instance, how is the projection function to satisfy coherency $cond(\tilde{x})$ defined? Since the features of the tabular datasets are often categorical or binary, it seems like using gradient descent is not the best approach to optimizing the perturbation. They have to introduce penalty terms to remain in the actional set $\mathcal{A}(x)$, and a projection function $cond(x)$ to satisfy coherency. There is not enough discussion on how effective this approach is and its limitations. I would suggest including a paragraph on the limitations and future work (scope for improvement) covering these aspects.

Author response:
We are working to improve the explanation of Algorithm 1 in our revised version.

Reviewer comment (Experiments):
The paper applies the Carlini-Wagner $\ell_2$ attack to generate adversarial inputs. However, it does not seem appropriate to apply this attack for these tabular datasets where some of the features are categorical or binary. It is likely to find adversarial inputs that are easily detectable by the verification function.

Author response:
We understand your concerns about the use of the Carlini-Wagner attack for a tabular data set, however, the field of adversarial attacks on tabular data is in its infancy with seminal work [1].  We were unable to find any trusted implementation of an adversarial attack on tabular data, so we decided that Carlini-Wagner was our best option.

Reviewer comment (Writing):
Minor: needs some proof-reading for language and typos. Some details about the algorithm are missing and could be included. Please see my suggestions in the `Questions` section.

Author response:
We are very grateful for your help and suggestions for improving language and catching typos.

Reviewer comment (Presentation):
The citation format of references is not correct. There should be a parentheses around citations; e.g. (Leo et al., 2019) instead of Leo et al., 2019.

Author response:
**Question**: The citation format of references is not correct. There should be a parentheses around citations; e.g. (Leo et al., 2019) instead of Leo et al., 2019.

**Answer**:  You are correct.  We are correcting that error in our updated version.

Reviewer comment (Writing):
The paper needs some proofreading for language issues and typos. For instance, in the last paragraph of page 2, it should be: 
> “In Section 2, we define an amicable perturbation as well as contrast it with related work.”

Author response:
**Question**: The paper needs some proofreading for language issues and typos. For instance, in the last paragraph of page 2, it should be:

    “In Section 2, we define an amicable perturbation as well as contrast it with related work.”


**Answer**:  You are correct.  We are currently proof reading our paper to improve readability and catch typos.  We have corrected the error you pointed out.

Reviewer comment (Writing):
The paper needs to define $k$ as the number of classes and $m$ as the input dimension at the start of Section 2. Sometimes these two symbols are interchanged leading to confusion. A few instances are listed below:
- In Eqn 28, it should be $m$ instead of $k$ denoting the number of features.
- In the proof of theorem 1 (Appendix 5.1), the number of classes is denoted by $m$ instead of $k$. 
- On page 6, under `Solving Step 1`, the projection function should be defined as $\text{cond} : \mathrm{R}^m \mapsto \mathcal{X}$.

Author response:
**Question**:  The paper needs to define $k$ as the number of classes and $m$ as the input dimension at the start of Section 2. Sometimes these two symbols are interchanged leading to confusion. A few instances are listed below:

-In Eqn 28, it should be $m$ instead of $k$ denoting the number of features.

-In the proof of theorem 1 (Appendix 5.1), the number of classes is denoted by $m$ instead of $k$.

-On page 6, under Solving Step 1, the projection function should be defined as $cond:\mathbb{R}^m\rightarrow\mathcal{X}$.

**Answer**:  Thank you for catching those errors.  We have made those corrections in our updated paper.

Reviewer comment (Writing):
In Section 2, Problem setting and goals, the notations could be more clear. Rather than saying $(\mathcal{X}, \mathcal{Y}) \sim \mathcal{D}$, it would be better to say $(x, y) \sim \mathcal{D}$, where $x \in \mathcal{X} \subset \mathrm{R}^m$ and $y \in \mathcal{Y}$. Here, $\mathcal{Y}$ is the $k$ probability simplex. Also, different from conventional notation where $y$ is an integer, here it is the true (conditional) probability distribution over the set of classes. Therefore, it would be clearer to define a class random variable $C \in \\{1, \cdots, k\\}$ and $y := [P(C=1 | x), \cdots, P(C=k | x)]$, i.e. the true class posterior probabilities. Of course, this could specialize to a one-hot coded label. With this definition, it is clear that an amicable perturbation $\tilde{x}$ aims to change the true class of an input to a desired $\tilde{y} := [P(C=1 | \tilde{x}), \cdots, P(C=k | \tilde{x})]$. In the definition of the difference training data (before Eqn 3), used to train $V(x, \tilde{x})$, it should not include pairs $(i, j)$ where $i = j$. If the architecture of $V$ does not depend on the order of inputs, then it should only include pairs where $j > i$. In Eqn (3), it would be simpler to define $z^{(i,j)} = \mathrm{1}[y^{(i)} = y^{(j)}]$, i.e. using the indicator function.

Author response:
**Question**:  Section 2, Problem setting and goals

The notations could be more clear. Rather than saying $(\mathcal{X},\mathcal{Y})\sim \mathcal{D}$, it would be better to say $(\mathbf{x},\mathbf{y})\sim \mathcal{D}$, where $\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^m$ and $\mathbf{y}\in\mathcal{Y}$. Here, $\mathcal{Y}$ is the $k$ probability simplex.

Also, different from conventional notation where $\mathbf{y}$ is an integer, here it is the true (conditional) probability distribution over the set of classes. Therefore, it would be clearer to define a class random variable $C\in\{1,...,k\}$ and $\mathbf{y}:=[\mathbb{P}(C=1|\mathbf{x}),...,\mathbb{P}(C=k|\mathbf{x})]$, i.e. the true class posterior probabilities. Of course, this could specialize to a one-hot coded label.

With this definition, it is clear that an amicable perturbation $\tilde{mathbf{x}}$ aims to change the true class of an input to a desired $\tilde{\mathbf{y}}:=[\mathbb{P}(C=1|\tilde{\mathbf{x}}),...,\mathbb{P}(C=k|\tilde{\mathbf{x}})]$.

In the definition of the difference training data (before Eqn 3), used to train $V(\mathbf{x},\tilde{\mathbf{x}})$, it should not include pairs $(i,j)$ where $i=j$. If the architecture of $V$ does not depend on the order of inputs, then it should only include pairs where $i>j$.

In Eqn (3), it would be simpler to define $z^{(i,j)} = 1[\mathbf{y}^{(i)} = \mathbf{y}^{(j)}]$, i.e. using the indicator function. 

**Answer**:  We appreciate your suggestions for improving clarity. We are looking into updating those notations for clarity while still maintaining consistent notations with other papers in this field.

Reviewer comment (Theory):
On training the verification function (page 4), the verification function $V$ is actually estimating the conditional probability that the true class corresponding to $x$ and $\tilde{x}$ are equal given the inputs. Let $C$ and $\tilde{C}$ denote the true class corresponding to $x$ and $\tilde{x}$. The verification function estimates the probability $P(C = \tilde{C} | x, \tilde{x})$. Another way to estimate this probability is using the classifier $M(x)$, assuming conditional independence of $C$ and $\tilde{C}$ given $(x, \tilde{x})$, as follows: 
$\hat{P}(C = \tilde{C} | x, \tilde{x}) = \sum_{i=1}^k M_i(x) M_i(\tilde{x})$

Author response:
**Question**:  The verification function is actually estimating the conditional probability that the true class corresponding to $\mathbf{x}$ and $\tilde{\mathbf{x}}$ are equal given the inputs.  Let $C$ and $\tilde{C}$ denote the true class corresponding  to $\mathbf{x}$ and $\tilde{\mathbf{x}}$. The verification function estimates the probability $\mathbb{P}(C,\tilde{C}|\tilde{\mathbf{x}})$. Another way to estimate this probability is using the classifier $M(\mathbf{x})$, assuming conditional independence of $C$ and $\tilde{C}$ given $(\mathbf{x},\tilde{\mathbf{x}})$, as follows: $\hat{\mathbb{P}}(C,\tilde{C}|\tilde{\mathbf{x}},\mathbf{x}) = \sum_{i=1}^k M_i(\mathbf{x})M_i(\tilde{\mathbf{x}})$

**Answer**:  We are trying to make this explanation more clear in our revised version and appreciate your suggestions on how we could do this.

Reviewer comment (Reproducibility):
Regarding Algorithm 1, would suggest defining the penalty terms $b(\tilde{x})$, $p(\tilde{x})$ and the projection function $cond(x)$. Is it possible to provide a general form for different scenarios? It would be good to provide pointers in the main paper to the penalty terms defined in Appendix 5.2.1. In the step where the gradient $\textbf{g}$ is calculated, please show that the gradient is over all the terms by including parentheses around the terms. In practice, is it sufficient to use a fixed learning rate $\alpha$ for the gradient descent? Have the authors explored adaptive learning rate schemes? Please add comments for some of the lines in the algorithm for clarity. For example, can add a comment like “// Projection to ensure coherancy” to the line $\tilde{x} = cond(\tilde{x})$. It would help to provide a precise method (recipe) to adjust $\lambda$ and the problem parameters. What is the method used in your implementation?

Author response:
**Question**:   Regarding Algorithm 1

Would suggest defining the penalty terms $b(\tilde{\mathbf{x}})$, $p(\tilde{\mathbf{x}})$ and the projection function $cond(\tilde{\mathbf{x}})$. Is it possible to provide a general form for different scenarios? It would be good to provide pointers in the main paper to the penalty terms defined in Appendix 5.2.1.

In the step where the gradient $\mathbf{g}$ is calculated, please show that the gradient is over all the terms by including parentheses around the terms.

In practice, is it sufficient to use a fixed learning rate $\alpha$ for the gradient descent? Have the authors explored adaptive learning rate schemes?

Please add comments for some of the lines in the algorithm for clarity. For example, can add a comment like “// Projection to ensure coherency” to the line $\tilde{\mathbf{x}} = cond(\tilde{\mathbf{x}})$.

It would help to provide a precise method (recipe) to adjust
and the problem parameters. What is the method used in your implementation?

**Answer**:  Thank you for your feedback.  We are attempting to clarify our description of Algorithm 1 based off of your recommendations.  In practice we used the ADAM modification of gradient descent.  We will mention that fact in the updated version.

Reviewer comment (Experiments):
In the experimental setup (Section 4), the cost function for each dataset should be $d_{\mathcal{X}}$, not $d_{\mathcal{Y}}$. Same comment for Figure 3.

Author response:
**Question**: In the experimental setup (Section 4), the cost function for each data set should be $d_\mathcal{X}$, not $d_\mathcal{Y}$. Same comment for Figure 3. 

**Answer**:  You are correct.  We have corrected those errors in our updated version.

Reviewer comment (Reproducibility):
In Section 4, the cost functions $d_{\mathcal{X}}$ defined for the datasets are somewhat vague. Could the authors describe them more precisely in an appendix?

Author response:
**Question**: In Section 4, the cost functions $d_\mathcal{X}$ defined for the data sets are somewhat vague. Could the authors describe them more precisely in an appendix?

**Answer**: We are adding a section to the appendix that includes more details on the cost functions.  We also note that the exact code for the cost functions is found in the supplementary material.

Reviewer comment (Writing):
The following lines under `Other Methods` in Section 4 is not clear.
> The counterfactuals will belong to the same actionable set as the amicable perturbations, but the adversarial examples not be be in the actionable set or even to be coherent. This is significant because our verifier procedure should be able to recognize that these adversarial examples are not effective real world solutions.

Author response:
**Question**: The following lines under Other Methods in Section 4 is not clear.

    The counterfactuals will belong to the same actionable set as the amicable perturbations, but the adversarial examples not be be in the actionable set or even to be coherent. This is significant because our verifier procedure should be able to recognize that these adversarial examples are not effective real world solutions.


**Answer**: We will clarify those lines in the updated version.  They are meant to explain that we have the same actionable set requirements for the counterfactuals as we do for the amicable perturbations.  The Carlini-Wagner attack, however, does not allow for the limiting of the set within which the adversarial examples are found.  Hence the adversarial examples may lie outside the actionable set and and also break coherency (formatting) rules.

Reviewer comment (Presentation):
Minor: the quality of math symbols in the figures can be improved.

Author response:
**Question**:  Minor: the quality of math symbols in the figures can be improved. 


**Answer**: We will look into ways to decrease pixelation.

Reviewer comment (Evaluation):
A discussion of limitations and societal impacts of amicable perturbations would be useful. Can include points such as the computational cost of generating amicable perturbations, which may not be so bad in the real world because it is not usually time sensitive. The verification method for detecting inputs could potentially be improved to reduce the false rejection of amicable perturbations as adversarial.

Author response:
**Question**: A discussion of limitations and societal impacts of amicable perturbations would be useful. Can include points such as the computational cost of generating amicable perturbations, which may not be so bad in the real world because it is not usually time sensitive. The verification method for detecting inputs could potentially be improved to reduce the false rejection of amicable perturbations as adversarial. 

**Answer**:  We will include those ideas in the conclusion of our revised paper.