Original: A NALOGIES
Revised: A NALOGIES

Original: AND
Revised: AND

Original: F EATURE
Revised: F EATURE

Original: A TTRIBUTIONS
Revised: A TTRIBUTIONS

Original: FOR
Revised: FOR

Original: M ODEL
Revised: M ODEL

Original: A GNOSTIC
Revised: A GNOSTIC

Original: E XPLANATION
Revised: E XPLANATION

Original: OF
Revised: OF

Original: S IMILARITY
Revised: S IMILARITY

Original: L EARNERS
Revised: L EARNERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Post-hoc explanations for black box models have been studied extensively in classiﬁcation and regression settings.
Revised: Post-hoc explanations for black box models have been studied extensively in classiﬁcation and regression settings.

Original: However, explanations for models that output similarity between two inputs have received comparatively lesser attention.
Revised: However, explanations for models that output similarity between two inputs have received comparatively lesser attention.

Original: In this paper, we provide model agnostic local explanations for similarity learners applicable to tabular and text data.
Revised: In this paper, we provide model agnostic local explanations for similarity learners applicable to tabular and text data.

Original: We ﬁrst propose a method that provides feature attributions to explain the similarity between a pair of inputs as determined by a black box similarity learner.
Revised: We ﬁrst propose a method that provides feature attributions to explain the similarity between a pair of inputs as determined by a black box similarity learner.

Original: We then propose analogies as a new form of explanation in machine learning.
Revised: We then propose analogies as a new form of explanation in machine learning.

Original: Here the goal is to identify diverse analogous pairs of examples that share the same level of similarity as the input pair and provide insight into (latent) factors underlying the model’s prediction.
Revised: Here the goal is to identify diverse analogous pairs of examples that share the same level of similarity as the input pair and provide insight into (latent) factors underlying the model’s prediction.

Original: The selection of analogies can optionally leverage feature attributions, thus connecting the two forms of explanation while still maintaining complementarity.
Revised: The selection of analogies can optionally leverage feature attributions, thus connecting the two forms of explanation while still maintaining complementarity.

Original: We prove that our analogy objective function is submodular, making the search for good-quality analogies efﬁcient.
Revised: We prove that our analogy objective function is submodular, making the search for good-quality analogies efﬁcient.

Original: We apply the proposed approaches to explain similarities between sentences as predicted by a state-of-the-art sentence encoder, and between patients in a healthcare utilization application.
Revised: We apply the proposed approaches to explain similarities between sentences as predicted by a state-of-the-art sentence encoder, and between patients in a healthcare utilization application.

Original: Efﬁcacy is measured through quantitative evaluations, a careful user study, and examples of explanations.
Revised: Efﬁcacy is measured through quantitative evaluations, a careful user study, and examples of explanations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The goal of a similarity function is to quantify the similarity between two objects.
Revised: The goal of a similarity function is to quantify the similarity between two objects.

Original: The learning of similarity functions from labeled examples, or equivalently distance functions, has traditionally been studied within the area of similarity or metric learning (Kulis, 2013).
Revised: The learning of similarity functions from labeled examples, or equivalently distance functions, has traditionally been studied within the area of similarity or metric learning (Kulis, 2013).

Original: With the advent of deep learning, learning complex similarity functions has found its way into additional important applications such as health care informatics, face recognition, handwriting analysis/signature veriﬁcation, and search engine query matching.
Revised: With the advent of deep learning, learning complex similarity functions has found its way into additional important applications such as health care informatics, face recognition, handwriting analysis/signature veriﬁcation, and search engine query matching.

Original: For example, learning pairwise similarity between patients in Electronic
Revised: For example, learning pairwise similarity between patients in Electronic

Original: Health Records (EHR) helps doctors in diagnosing and treating future patients (Zhu et al., 2016).
Revised: Health Records (EHR) helps doctors in diagnosing and treating future patients (Zhu et al., 2016).

Original: Although deep similarity models may better quantify similarity, the complexity of these models could make them harder to trust.
Revised: Although deep similarity models may better quantify similarity, the complexity of these models could make them harder to trust.

Original: For decision-critical systems like patient diagnosis and treatment, it would be helpful for users to understand why a black box model assigns a certain level of similarity to two objects.
Revised: For decision-critical systems like patient diagnosis and treatment, it would be helpful for users to understand why a black box model assigns a certain level of similarity to two objects.

Original: Providing explanations for similarity models is therefore an important problem.
Revised: Providing explanations for similarity models is therefore an important problem.

Original: Explanations for ML models have been studied extensively in classiﬁcation and regression settings.
Revised: Explanations for ML models have been studied extensively in classiﬁcation and regression settings.

Original: Local explanations in particular have received a lot of attention (Ribeiro et al., 2016; Lundberg & Lee, 2017; Selvaraju et al., 2020) given that entities (viz.
Revised: Local explanations in particular have received a lot of attention (Ribeiro et al., 2016; Lundberg & Lee, 2017; Selvaraju et al., 2020) given that entities (viz.

Original: individuals) are primarily interested in understanding why a certain decision was made for them, and building globally interpretable surrogates for a black box model is typically much more challenging.
Revised: individuals) are primarily interested in understanding why a certain decision was made for them, and building globally interpretable surrogates for a black box model is typically much more challenging.

Original: Local explanations can uncover potential issues such as reliance on unimportant or unfair factors in a region of the input space, hence aiding in debugging a black box model.
Revised: Local explanations can uncover potential issues such as reliance on unimportant or unfair factors in a region of the input space, hence aiding in debugging a black box model.

Original: Appropriately aggregating local explanations can also provide reasonable global understanding (van der Linden et al., 2019; Ramamurthy et al., 2020).
Revised: Appropriately aggregating local explanations can also provide reasonable global understanding (van der Linden et al., 2019; Ramamurthy et al., 2020).

Original: In this paper, we develop model-agnostic local explanation methods for similarity learners, which is a relatively under-explored area.
Revised: In this paper, we develop model-agnostic local explanation methods for similarity learners, which is a relatively under-explored area.

Original: Given a black box similarity learner and a pair of inputs, our ﬁrst method produces feature attributions for the output of the black box.
Revised: Given a black box similarity learner and a pair of inputs, our ﬁrst method produces feature attributions for the output of the black box.

Original: We discuss why the direct application of LIME (Ribeiro et al., 2016) and other ﬁrst-order methods is less satisfactory for similarity models.
Revised: We discuss why the direct application of LIME (Ribeiro et al., 2016) and other ﬁrst-order methods is less satisfactory for similarity models.

Original: We then propose a quadratic approximation using
Revised: We then propose a quadratic approximation using

Original: Mahalanobis distance.
Revised: Mahalanobis distance.

Original: A simpliﬁed example of the output is shown as shading in Figure 1.
Revised: A simpliﬁed example of the output is shown as shading in Figure 1.

Original: [Table]
Revised: [Table]

Original: Our second contribution is to propose a novel type of explanation in the form of analogies for a given input pair.
Revised: Our second contribution is to propose a novel type of explanation in the form of analogies for a given input pair.

Original: The importance of analogybased explanations was recently advocated for by (Hullermeier, 2020).
Revised: The importance of analogybased explanations was recently advocated for by (Hullermeier, 2020).

Original: We formalize this with an objective function that captures the intuitive desiderata of (1) closeness in degree of similarity to the input pair, (2) diversity among the analogous pairs, and (3) a notion of analogy, i.e., members of each analogous pair have a similar relationship to each other as members of the input pair.
Revised: We formalize this with an objective function that captures the intuitive desiderata of (1) closeness in degree of similarity to the input pair, (2) diversity among the analogous pairs, and (3) a notion of analogy, i.e., members of each analogous pair have a similar relationship to each other as members of the input pair.

Original: We prove that this objective is submodular, making it efﬁcient to ﬁnd good analogies within a large dataset.
Revised: We prove that this objective is submodular, making it efﬁcient to ﬁnd good analogies within a large dataset.

Original: An example analogy is shown in Figure 1.
Revised: An example analogy is shown in Figure 1.

Revised: The analogy is understandable since, in the input one of the sentences provides more context (i.e. presence of a fraudulent scheme called “dolphin scheme"), similar to the analogy where Singapore being a “small city-state" is the additional context.

Original: The example suggests that analogies can uncover appropriate (latent) factors to explain predictions, which may not be apparent from explicit features such as words/phrases.
Revised: This thus suggests that analogies can uncover appropriate (latent) factors to explain predictions, which may not be apparent from explicit features such as words/phrases.

Original: The proposed feature- and analogy-based methods are applied to text and tabular data, to explain similarities between i) sentences from the Semantic Textual Similarity (STS) dataset (Cer et al., 2017), ii) patients in terms of their healthcare utilization using Medical Expenditure Panel Survey (MEPS) data, and iii) iris species (IRIS).
Revised: The proposed feature- and analogy-based methods are applied to text and tabular data, to explain similarities between i) sentences from the Semantic Textual Similarity (STS) dataset (Cer et al., 2017), ii) patients in terms of their healthcare utilization using Medical Expenditure Panel Survey (MEPS) data, and iii) iris species (IRIS).

Original: The proposed methods outperform feature- and exemplar-based baselines in both the quantitative evaluation and human user study we conduct, showing high ﬁdelity to the black box similarity learner and providing reasons that users ﬁnd sensible.
Revised: The proposed methods outperform featureand exemplar-based baselines in both the quantitative evaluation and human user study we conduct, showing high ﬁdelity to the black box similarity learner and providing reasons that users ﬁnd sensible.

Original: We also presentin-depth examples of feature- and analogy-based explanations and illustrate speciﬁc insights.
Revised: We also present examples of feature- and analogy-based explanations and illustrate speciﬁc insights.

Original:  P ROBLEM D ESCRIPTION
Revised:  P ROBLEM D ESCRIPTION

Original: Given a pair of examples x = ( x 1 , x 2 ) ∈
Revised: Given a pair of examples x = ( x 1 , x 2 ) ∈

Original: R m ⊗ R m , where m is the dimension of the space, and a black box model δ BB ( . ) :
Revised: R m ⊗ R m , where m is the dimension of the space, and a black box model δ BB ( . ) :

Original: R m ⊗ R m (cid:55)→ R , our goal is to “explain” the prediction δ BB ( x ) of the black box model.
Revised: R m ⊗ R m (cid:55)→ R , our goal is to “explain” the prediction δ BB ( x ) of the black box model.

Original: One type of explanation takes the form of a sparse set of features (i.e. (cid:28) m if m is large) that are most important in determining the output, together possibly with weights to quantify their importance.
Revised: One type of explanation takes the form of a sparse set of features (i.e. (cid:28) m if m is large) that are most important in determining the output, together possibly with weights to quantify their importance.

Original: An alternative form of explanation consists of other example pairs that have the same (or similar) output from the black box model as the input pair.
Revised: An alternative form of explanation consists of other example pairs that have the same (or similar) output from the black box model as the input pair.

Original: The latter constitutes a new form of (local) explanation which we term as analogy-based explanation .
Revised: The latter constitutes a new form of (local) explanation which we term as analogy-based explanation .

Original: Although these might seem to be similar to exemplar-based explanations (Gurumoorthy et al., 2019), which are commonly used to locally explain classiﬁcation models, there is an important difference: Exemplars are typically close to the inputs they explain, whereas analogies do not have to be.
Revised: Although these might seem to be similar to exemplar-based explanations (Gurumoorthy et al., 2019), which are commonly used to locally explain classiﬁcation models, there is an important difference: Exemplars are typically close to the inputs they explain, whereas analogies do not have to be.

Original: What is desired is for the relationship between members of each analogous pair to be close to the relationship of the input pair ( x 1 , x 2 ) .
Revised: What is desired is for the relationship between members of each analogous pair to be close to the relationship of the input pair ( x 1 , x 2 ) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A brief survey of local and global explanation methods are available in Appendix A.
Revised: A brief survey of local and global explanation methods are available in Appendix A.

Original: We are aware of only a few works that explain similarity models (Zheng et al., 2020; Plummer et al., 2020; Zhu et al., 2021).
Revised: We are aware of only a few works that explain similarity models (Zheng et al., 2020; Plummer et al., 2020; Zhu et al., 2021).

Original: All of these however are primarily applicable to image data.
Revised: All of these however are primarily applicable to image data.

Original: Moreover, they either require white-box access or are based on differences between saliency maps.
Revised: Moreover, they either require white-box access or are based on differences between saliency maps.

Original: Our methods on the other hand are model agnostic and apply to tabular and text data, as showcased in the experiments.
Revised: Our methods on the other hand are model agnostic and apply to tabular and text data, as showcased in the experiments.

Original: The Joint Search LIME (JSLIME) method, proposed in (Hamilton et al., 2021) for model-agnostic explanations of image similarity, has parallels to our feature-based explanations.
Revised: The Joint Search LIME (JSLIME) method, proposed in (Hamilton et al., 2021) for model-agnostic explanations of image similarity, has parallels to our feature-based explanations.

Original: However, JSLIME is geared toward ﬁnding corresponding regions between a query and a retrieved image, whereas our feature-based method explains the distance predicted by a similarity model by another, simpler distance function and is more natural for tabular data (See Appendix E.1 for more details).
Revised: However, JSLIME is geared toward ﬁnding corresponding regions between a query and a retrieved image, whereas our feature-based method explains the distance predicted by a similarity model by another, simpler distance function and is more natural for tabular data (See Appendix E.1 for more details).

Original: There is a rich literature on similarity/metric learning methods, see e.g. (Kulis, 2013) for a survey.
Revised: There is a rich literature on similarity/metric learning methods, see e.g. (Kulis, 2013) for a survey.

Original: However, the goal in these works is to learn a global metric from labeled examples.
Revised: However, the goal in these works is to learn a global metric from labeled examples.

Original: The labels may take the form of real-valued similarities or distances (regression similarity learning) (Weinberger & Saul, 2009); binary similar/dissimilar labels (Tariq et al., 2020), which may come from set membership or consist of pairwise “must-link” or “cannot-link” constraints (Zhang et al., 2019); or triplets ( x, y, z ) where y is more similar to x than z (contrastive learning)
Revised: The labels may take the form of real-valued similarities or distances (regression similarity learning) (Weinberger & Saul, 2009); binary similar/dissimilar labels (Tariq et al., 2020), which may come from set membership or consist of pairwise “must-link” or “cannot-link” constraints (Zhang et al., 2019); or triplets ( x, y, z ) where y is more similar to x than z (contrastive learning)

Original: (Schultz & Joachims, 2004;
Revised: (Schultz & Joachims, 2004;

Original: Hoffer & Ailon, 2015).
Revised: Hoffer & Ailon, 2015).

Original: Importantly, the metric does not have to be interpretable, and recent deep learning models are certainly not.
Revised: Importantly, the metric does not have to be interpretable, and recent deep learning models are certainly not.

Original: In our setting, we are given a similarity function as a black box and wish to explain its outputs at a local level.
Revised: In our setting, we are given a similarity function as a black box and wish to explain its outputs at a local level.

Original: Hence, the two problems are distinct.
Revised: Hence, the two problems are distinct.

Original: Mathematically, the feature-based method of Section 4.1 belongs to the regression similarity learning category (Kar &
Revised: Mathematically, the feature-based method of Section 4.1 belongs to the regression similarity learning category (Kar &

Original: Jain, 2012), but the supervision comes from the given black box.
Revised: Jain, 2012), but the supervision comes from the given black box.

Original: Note that our notion of analogies is different from the area of analogy mining (Hope et al., 2017), where representations are learnt from datasets in order to retrieve information with a certain intent.
Revised: Note that our notion of analogies is different from the area of analogy mining (Hope et al., 2017), where representations are learnt from datasets in order to retrieve information with a certain intent.

Original:  E XPLANATION M ETHODS
Revised:  E XPLANATION M ETHODS

Original: We propose two methods to explain similarity learners.
Revised: We propose two methods to explain similarity learners.

Original: The ﬁrst is a feature-based explanation, while the second is a new type of explanation termed as analogy-based explanation.
Revised: The ﬁrst is a feature-based explanation, while the second is a new type of explanation termed as analogy-based explanation.

Original: The two explanations complement each other, while at the same time are also related as the analogy-based explanation can optionally use the output of the feature-based method as input pointing to synergies between the two.
Revised: The two explanations complement each other, while at the same time are also related as the analogy-based explanation can optionally use the output of the feature-based method as input pointing to synergies between the two.

Original: 4. F EATURE -B ASED S IMILARITY E XPLANATIONS
Revised: 4. F EATURE -B ASED S IMILARITY E XPLANATIONS

Original: We assume that the black box model δ BB ( x, y ) is a distance function between two points x and y , i.e., smaller δ BB ( x, y ) implies greater similarity.
Revised: We assume that the black box model δ BB ( x, y ) is a distance function between two points x and y , i.e., smaller δ BB ( x, y ) implies greater similarity.

Original: We do not assume that δ BB satisﬁes all four axioms of a metric , although the the proposed local approximation is a metric and may be more suitable if δ BB satisﬁes some of the axioms.
Revised: We do not assume that δ BB satisﬁes all four axioms of a metric , although the the proposed local approximation is a metric and may be more suitable if δ BB satisﬁes some of the axioms.

Original: Given the literature on post-hoc explanation of classiﬁers and regressors, a natural way to obtain a feature-based explanation of δ BB ( x, y ) is to regard it as a function of a single input – the concatenation of ( x, y ) .
Revised: Given the literature on post-hoc explanation of classiﬁers and regressors, a natural way to obtain a feature-based explanation of δ BB ( x, y ) is to regard it as a function of a single input – the concatenation of ( x, y ) .

Original: Then LIME (Ribeiro et al., 2016) or other ﬁrst-order gradient-based methods (Selvaraju et al., 2020) can produce a local linear approximation of δ BB ( x, y ) at ( x, y ) of the form
Revised: Then LIME (Ribeiro et al., 2016) or other ﬁrst-order gradient-based methods (Simonyan et al., 2013) can produce a local linear approximation of δ BB ( x, y ) at ( x, y ) of the form

Original: g Tx ∆ x + g Ty ∆ y .
Revised: g Tx ∆ x + g Ty ∆ y .

Original: This approach cannot create interactions and thus cannot provide explanations in terms of distances between elements of x and y , e.g. ( x j − y j ) 2 or | x j − y j | , which are necessarily nonlinear.
Revised: This approach cannot create interactions and thus cannot provide explanations in terms of distances between elements of x and y , e.g. ( x j − y j ) 2 or | x j − y j | , which are necessarily nonlinear.

Original: We thus propose to locally approximate δ BB ( x, y ) with a quadratic model, the Mahalanobis distance δ I ( x, y ) =
Revised: We thus propose to locally approximate δ BB ( x, y ) with a quadratic model, the Mahalanobis distance δ I ( x, y ) =

Original: (¯ x − ¯ y ) T A (¯ x − ¯ y ) , where A (cid:23) 0 is a positive semideﬁnite matrix and ¯ x , ¯ y are interpretable representations of x , y (see (Ribeiro et al., 2016) and note that ¯ x = x , ¯ y
Revised: (¯ x − ¯ y ) T A (¯ x − ¯ y ) , where A (cid:23) 0 is a positive semideﬁnite matrix and ¯ x , ¯ y are interpretable representations of x , y (see (Ribeiro et al., 2016) and note that ¯ x = x , ¯ y

Original: =
Revised: =

Original: y if the features in x , y are already interpretable).
Revised: y if the features in x , y are already interpretable).

Original: This simple, interpretable approximation is itself a distance between x and y .
Revised: This simple, interpretable approximation is itself a distance between x and y .

Original: In Appendix E.2, we discuss the equivalence between explaining distances and similarities.
Revised: In Appendix E.2, we discuss the equivalence between explaining distances and similarities.

Original: In
Revised: In

Original: Section 5.1, we show qualitative examples for how elements of learned A can explain similarities.
Revised: Section 5.1, we show qualitative examples for how elements of learned A can explain similarities.

Original: We learn A by minimizing the following loss over a set of perturbations ( x i , y i ) in the neighborhood
Revised: We learn A by minimizing the following loss over a set of perturbations ( x i , y i ) in the neighborhood

Original: N xy of the input pair ( x, y ) :
Revised: N xy of the input pair ( x, y ) :

Original: [Equation]
Revised: [Equation]

Original: The loss captures the ﬁdelity of the Mahalanobis approximation to the black box.
Revised: The loss captures the ﬁdelity of the Mahalanobis approximation to the black box.

Original: For non-negative weights w x
Revised: For non-negative weights w x

Original: i ,y i , problem equation 1 is convex because 1) the quadratic form (¯ x i − ¯ y i ) T A (¯ x i − ¯ y i ) is linear in A , 2) this is composed with a weighted least squares objective, and 3) the set of semideﬁnite matrices is convex.
Revised: i ,y i , problem equation 1 is convex because 1) the quadratic form (¯ x i − ¯ y i ) T A (¯ x i − ¯ y i ) is linear in A , 2) this is composed with a weighted least squares objective, and 3) the set of semideﬁnite matrices is convex.

Original: At the same time, the semideﬁnite constraint A (cid:60) 0 makes equation 1 different from LIME.
Revised: At the same time, the semideﬁnite constraint A (cid:60) 0 makes equation 1 different from LIME.

Original: We use CVXPY (Diamond & Boyd, 2016; Agrawal et al., 2018) to solve equation 1.
Revised: We use CVXPY (Diamond & Boyd, 2016; Agrawal et al., 2018) to solve equation 1.

Original: The generation of perturbations ( x i , y i ) and their weighting by w x i ,y i mostly follow LIME’s approach (and share its limitations (Slack et al., 2020)), with the following two differences:
Revised: The generation of perturbations ( x i , y i ) and their weighting by w x i ,y i mostly follow LIME’s approach (and share its limitations (Slack et al., 2020)), with the following two differences:

Original: First and most notably, for perturbing categorical features, we use a method based on conditional probability models that generates more realistic perturbations.
Revised: First and most notably, for perturbing categorical features, we use a method based on conditional probability models that generates more realistic perturbations.

Original: This is described in Appendix G along with other perturbation details.
Revised: This is described in Appendix G along with other perturbation details.

Original: Second, we compute the weights w x
Revised: Second, we compute the weights w x

Original: i ,y i as w x,x i + w y,y i , where w x,x i (similarly w y,y i ) is computed as in LIME by applying an exponential kernel to a distance between x and x i .
Revised: i ,y i as w x,x i + w y,y i , where w x,x i (similarly w y,y i ) is computed as in LIME by applying an exponential kernel to a distance between x and x i .

Original: To further ease interpretation of the Mahalanobis explanation, we also consider a version in which A is constrained to be diagonal.
Revised: To further ease interpretation of the Mahalanobis explanation, we also consider a version in which A is constrained to be diagonal.

Original: Here, the quadratic form can be simpliﬁed as (¯ x − ¯ y ) T A (¯ x − ¯ y )
Revised: Here, the quadratic form can be simpliﬁed as (¯ x − ¯ y ) T A (¯ x − ¯ y )

Original: = a T s where a = diag( A ) and s has components s j = (¯ x j − ¯ y j ) 2 .
Revised: = a T s where a = diag( A ) and s has components s j = (¯ x j − ¯ y j ) 2 .

Original: Further, the constraint A (cid:60) 0 reduces to a ≥ 0 simplifying problem (1) into least-squares regression with a non-negativity constraint.
Revised: Further, the constraint A (cid:60) 0 reduces to a ≥ 0 simplifying problem (1) into least-squares regression with a non-negativity constraint.

Original: 4. A NALOGY -B ASED S IMILARITY E XPLANATIONS
Revised: 4. A NALOGY -B ASED S IMILARITY E XPLANATIONS

Original: We now describe a method of providing analogies as local explanations for similarity learners.
Revised: We now describe a method of providing analogies as local explanations for similarity learners.

Original: Given an input example pair ( x, y ) and a black box model, the goal is to identify a set of diverse pairs of examples from the dataset that have the same (or similar) relationship to each other as the input pair.
Revised: Given an input example pair ( x, y ) and a black box model, the goal is to identify a set of diverse pairs of examples from the dataset that have the same (or similar) relationship to each other as the input pair.

Original: The diversity can help weed out less important factors that one might otherwise think are important.
Revised: The diversity can help weed out less important factors that one might otherwise think are important.

Original: For example, let us say two patients have similar disease conditions (input pair) based on which the model predicted them as being similar.
Revised: For example, let us say two patients have similar disease conditions (input pair) based on which the model predicted them as being similar.

Original: The analogous pairs can be other pairs of patients who are also similar to each other in their disease conditions, but are perhaps socio-economically diverse.
Revised: The analogous pairs can be other pairs of patients who are also similar to each other in their disease conditions, but are perhaps socio-economically diverse.

Original: This will help ascertain that disease conditions are the reason for the similarity and not socio-economic factors.
Revised: This will help ascertain that disease conditions are the reason for the similarity and not socio-economic factors.

Original: In other words, the true latent factors responsible for the black box’s prediction can be uncovered using our analogies .
Revised: In other words, the true latent factors responsible for the black box’s prediction can be uncovered using our analogies .

Original: This is not to say that our analogies can never be similar to the input pair, but that they will be so only if the true relationship is not obscured.
Revised: This is not to say that our analogies can never be similar to the input pair, but that they will be so only if the true relationship is not obscured.

Original: For instance, in Example 1 in section 5.1 the ﬁrst analogous pair is very similar to the input pair since the words describing the action (viz.
Revised: For instance, in Example 1 in section 5.1 the ﬁrst analogous pair is very similar to the input pair since the words describing the action (viz.

Original: playing) are more important than the object of the sentence (viz.
Revised: playing) are more important than the object of the sentence (viz.

Original: harp, keyboard).
Revised: harp, keyboard).

Original: As such, analogy based explanations can be seen as a more unbiased way of explaining requiring human judgement, than feature attributions where the reasons are directly provided, making the two somewhat complimentary.
Revised: As such, analogy based explanations can be seen as a more unbiased way of explaining requiring human judgement, than feature attributions where the reasons are directly provided, making the two somewhat complimentary.

Original: Nonetheless, as we will see next our analogy based explanations can take into account the feature attributions to the extent desired (see equation 3).
Revised: Nonetheless, as we will see next our analogy based explanations can take into account the feature attributions to the extent desired (see equation 3).

Original: Let pairs of examples in a dataset be z
Revised: Let pairs of examples in a dataset be z

Original: i =
Revised: i =

Original: ( z i 1 , z
Revised: ( z i 1 , z

Original: i 2 ) for i ∈ { 1 , ..., N } and an input instance pair be x = ( x 1 , x 2 ) .
Revised: i 2 ) for i ∈ { 1 , ..., N } and an input instance pair be x = ( x 1 , x 2 ) .

Original: Given a black box model δ BB ( . ) and an analogy closeness function G ( z i , x ) to be deﬁned, the goal is to ﬁnd k analogous pairs to x by solving the following for λ 1 , λ 2 ≥ 0 :
Revised: Given a black box model δ BB ( . ) and an analogy closeness function G ( z i , x ) to be deﬁned, the goal is to ﬁnd k analogous pairs to x by solving the following for λ 1 , λ 2 ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: where, δ min ( z i , z j ) = min (cid:2) δ BB (( z i 1 , z j 1 ))
Revised: where, δ min ( z i , z j ) = min (cid:2) δ BB (( z i 1 , z j 1 ))

Original: + δ BB (( z i 2 , z j 2 )) ,
Revised: + δ BB (( z i 2 , z j 2 )) ,

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in equation 2 ensures that the analogous pair z i chosen has a similar distance between its members
Revised: The ﬁrst term in equation 2 ensures that the analogous pair z i chosen has a similar distance between its members

Original: z i 1 , z i 2 as the input pair ( x 1 , x 2 ) , according to the black box.
Revised: z i 1 , z i 2 as the input pair ( x 1 , x 2 ) , according to the black box.

Original: The last term encourages diversity in the analogous pairs such that the individual instances are different across pairs, although the similarity/difference within a pair is close to that of the input.
Revised: The last term encourages diversity in the analogous pairs such that the individual instances are different across pairs, although the similarity/difference within a pair is close to that of the input.

Original: The function δ min ( z i , z j ) determines the best matching between two pairs ( z i , z j ) .
Revised: The function δ min ( z i , z j ) determines the best matching between two pairs ( z i , z j ) .

Original: For the analogy closeness term G ( z i , x ) , we use
Revised: For the analogy closeness term G ( z i , x ) , we use

Original: [Equation]
Revised: [Equation]

Original: In equation 3, δ I ( x ) =
Revised: In equation 3, δ I ( x ) =

Original: (¯ x 1 − ¯ x 2 ) T A (¯ x 1 − ¯ x 2 ) is the distance predicted by the feature-based explanation of Section 4.1.
Revised: (¯ x 1 − ¯ x 2 ) T A (¯ x 1 − ¯ x 2 ) is the distance predicted by the feature-based explanation of Section 4.1.

Original: The inclusion of this term with weight α > 0 may be helpful if the feature-based explanation is faithful and we wish to directly interpret the analogies.
Revised: The inclusion of this term with weight α > 0 may be helpful if the feature-based explanation is faithful and we wish to directly interpret the analogies.

Original: The term D ( z i , x ) is the cosine distance between the directions φ ( z i 2 ) − φ
Revised: The term D ( z i , x ) is the cosine distance between the directions φ ( z i 2 ) − φ

Original: ( z i 1 ) and φ ( x 2 ) − φ ( x 1 ) in an embedding space.
Revised: ( z i 1 ) and φ ( x 2 ) − φ ( x 1 ) in an embedding space.

Original: Here φ is an embedding function that can be the identity or chosen independently of the black box, hence preserving the model-agnostic nature of the interpretations.
Revised: Here φ is an embedding function that can be the identity or chosen independently of the black box, hence preserving the model-agnostic nature of the interpretations.

Original: The intuition is that these directions capture aspects of the relationships between z i 1 , z
Revised: The intuition is that these directions capture aspects of the relationships between z i 1 , z

Original: i 2 and between x 1 , x 2 .
Revised: i 2 and between x 1 , x 2 .

Original: We will hence refer to this as direction similarity .
Revised: We will hence refer to this as direction similarity .

Original: In summary, the terms in equation 2—equation 4 together are aimed at producing faithful, intuitive and diverse analogies as explanations.
Revised: In summary, the terms in equation 2—equation 4 together are aimed at producing faithful, intuitive and diverse analogies as explanations.

Original: Let f ( { z 1 , . .
Revised: Let f ( { z 1 , . .

Original: z k } ) denote the objective in equation 2.
Revised: z k } ) denote the objective in equation 2.

Original: We prove the following in Appendix B.
Revised: We prove the following in Appendix B.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: The objective function f in (equation 2) is submodular.
Revised: The objective function f in (equation 2) is submodular.

Original: Given that our function is submodular, we can use well-known minimization methods to ﬁnd a k -sparse solution with approximation guarantees (Svitkina & Fleischer, 2011).
Revised: Given that our function is submodular, we can use well-known minimization methods to ﬁnd a k -sparse solution with approximation guarantees (Svitkina & Fleischer, 2011).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present ﬁrst in Section 5.1 examples of explanations obtained with our proposed methods, to illustrate insights that may be derived.
Revised: We present ﬁrst in Section 5.1 examples of explanations obtained with our proposed methods, to illustrate insights that may be derived.

Original: Our formal experimental study consists of both a human evaluation to investigate the utility of different explanations (Section 5.2) as well as quantitative analysis (Section 5.3).
Revised: Our formal experimental study consists of both a human evaluation to investigate the utility of different explanations (Section 5.2) as well as quantitative analysis (Section 5.3).

Original: The numerical experiments were run with embarrassing parallelization on acore/64 GB RAM Linux machine.
Revised: The numerical experiments were run with embarrassing parallelization on acore/64 GB RAM Linux machine.

Original: 56 cores and 242 GB of RAM were used for larger experiments.
Revised: 56 cores and 242 GB of RAM were used for larger experiments.

Original: 5. Q UALITATIVE E XAMPLES
Revised: 5. Q UALITATIVE E XAMPLES

Original: We discuss examples of the proposed feature-based explanations with full A matrix (FbFull) and analogy-based explanations (AbE).
Revised: We discuss examples of the proposed feature-based explanations with full A matrix (FbFull) and analogy-based explanations (AbE).

Original: The examples are taken from the Semantic Textual Similarity (STS) benchmark dataset 1 (Cer et al., 2017) described further below.
Revised: The examples are taken from the Semantic Textual Similarity (STS) benchmark dataset 1 (Cer et al., 2017) described further below.

Original: STS dataset: The dataset has 8628 sentence pairs, divided into training, validation, and test sets.
Revised: STS dataset: The dataset has 8628 sentence pairs, divided into training, validation, and test sets.

Original: Each pair has a ground truth semantic similarity score that we convert to a distance.
Revised: Each pair has a ground truth semantic similarity score that we convert to a distance.

Original: For the black box similarity model δ BB ( x, y ) , we use the cosine distance between the embeddings of x and y produced by the universal sentence encoder 2 (Cer et al., 2018a).
Revised: For the black box similarity model δ BB ( x, y ) , we use the cosine distance between the embeddings of x and y produced by the universal sentence encoder 2 (Cer et al., 2018a).

Original: It is possible to learn a distance on top of these embeddings, but we ﬁnd that the Pearson correlation of 0 .
Revised: It is possible to learn a distance on top of these embeddings, but we ﬁnd that the Pearson correlation of 0 .

Original: 787 between the cosine distances and true distances is already competitive with the STS benchmarks (Wang et al., 2019).
Revised: 787 between the cosine distances and true distances is already competitive with the STS benchmarks (Wang et al., 2019).

Original: The corresponding mean absolute error is 0 .
Revised: The corresponding mean absolute error is 0 .

Original: In any case, our methods are agnostic to the black box model.
Revised: In any case, our methods are agnostic to the black box model.

Original: AbE hyperparameters:
Revised: AbE hyperparameters:

Original: In all experiments, we set α = 0 to assess the value of AbE independent of feature-based explanations.
Revised: In all experiments, we set α = 0 to assess the value of AbE independent of feature-based explanations.

Original: λ 

Revised: λ

Original: and λ 2 were selected once per dataset (not tuned per example) by evaluating the average ﬁdelity of the analogies to the input pairs in terms of the black box model’s predictions, along with manually inspecting a random subset of analogies to see how intuitive they were.
Revised: and λ 2 were selected once per dataset (not tuned per example) by evaluating the average ﬁdelity of the analogies to the input pairs in terms of the black box model’s predictions, along with manually inspecting a random subset of analogies to see how intuitive they were.

Original: With STS, we get λ 1 = 0 .
Revised: With STS, we get λ 1 = 0 .

Original: 5 , λ 2 = 0 .
Revised: 5 , λ 2 = 0 .

Original: Analogies with other baselines are in Appendix K.
Revised: Analogies with other baselines are in Appendix K.

Original: Example 1: To start with a simple example, we consider the following pair of sentences.
Revised: Example 1: To start with a simple example, we consider the following pair of sentences.

Original: (a) A man is playing a harp.
Revised: (a) A man is playing a harp.

Original: (b) A man is playing a keyboard.
Revised: (b) A man is playing a keyboard.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: This pair was assigned a distance of 0 .
Revised: This pair was assigned a distance of 0 .

Original: 38 by the black box (BB) similarity model.
Revised: 38 by the black box (BB) similarity model.

Original: FbFull approximates the above distance by the Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) .
Revised: FbFull approximates the above distance by the Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) .

Original: For STS, the interpretable representation ¯ x is a binary vector with each component ¯ x j indicating whether a word is present in the sentence.
Revised: For STS, the interpretable representation ¯ x is a binary vector with each component ¯ x j indicating whether a word is present in the sentence.

Original: We deﬁne C to be the distance contribution matrix C whose elements C jk :=
Revised: We deﬁne C to be the distance contribution matrix C whose elements C jk :=

Original: (¯ x j − ¯ y j )
Revised: (¯ x j − ¯ y j )

Original: A jk (¯ x k − ¯ y k ) sum up to the Mahalanobis distance.
Revised: A jk (¯ x k − ¯ y k ) sum up to the Mahalanobis distance.

Original: The distance contributions C jk for
Revised: The distance contributions C jk for

Original: Example 1 are shown in Figure 2a.
Revised: Example 1 are shown in Figure 2a.

Original: Since the substitution of “keyboard” for “harp” is the only difference between the sentences, these are the only rows/columns with non-zero entries.
Revised: Since the substitution of “keyboard” for “harp” is the only difference between the sentences, these are the only rows/columns with non-zero entries.

Original: A diagonal element C jj is the contribution due to one sentence having word j and the other lacking it (e.g. ¯ x j = 1 , ¯ y j = 0 ).
Revised: A diagonal element C jj is the contribution due to one sentence having word j and the other lacking it (e.g. ¯ x j = 1 , ¯ y j = 0 ).

Original: Interestingly, the diagonal elements are partially offset by negative off-diagonal elements
Revised: Interestingly, the diagonal elements are partially offset by negative off-diagonal elements

Original: C jk , which represent a contribution due to substituting word j ( ¯ x j = 1 , ¯ y j = 0 ) for word k ( ¯ x k = 0 , ¯ y k = 1 ).
Revised: C jk , which represent a contribution due to substituting word j ( ¯ x j = 1 , ¯ y j = 0 ) for word k ( ¯ x k = 0 , ¯ y k = 1 ).

Original: We can presume that the offset occurs because harp and keyboard are both musical instruments and thus somewhat similar.
Revised: We can presume that the offset occurs because harp and keyboard are both musical instruments and thus somewhat similar.

Original: AbE gives the following top three analogies:
Revised: AbE gives the following top three analogies:

Original: (a) A guy is playing hackysack.
Revised: (a) A guy is playing hackysack.

Original: (b) A man is playing a key-board.
Revised: (b) A man is playing a key-board.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: (a) Women are running.
Revised: (a) Women are running.

Original: (b) Two women are running.
Revised: (b) Two women are running.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: (a) There’s no rule that decides which players can be picked for bowling/batting in the Super Over.
Revised: (a) There’s no rule that decides which players can be picked for bowling/batting in the Super Over.

Original: (b) Yes a team can use the same player for both bowling and batting in a super over.
Revised: (b) Yes a team can use the same player for both bowling and batting in a super over.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: The ﬁrst analogy is very similar except that hackysack is a sport rather than a musical instrument.
Revised: The ﬁrst analogy is very similar except that hackysack is a sport rather than a musical instrument.

Original: The sentences in the second pair are more similar than the input pair as reﬂected in the corresponding
Revised: The sentences in the second pair are more similar than the input pair as reﬂected in the corresponding

Original: BB distance.
Revised: BB distance.

Original: The third analogy is less related (both sentences are about cricket player selection) with a larger BB distance.
Revised: The third analogy is less related (both sentences are about cricket player selection) with a larger BB distance.

Original: Example 2: Next we consider the pair of longer sentences from Figure 1.
Revised: Example 2: Next we consider the pair of longer sentences from Figure 1.

Original: The BB distance between this pair is 0 .
Revised: The BB distance between this pair is 0 .

Original: so they are closer than in Example 1.
Revised: so they are closer than in Example 1.

Original: The two sentences are mostly the same but the ﬁrst one adds context about an additional dolphin scheme.
Revised: The two sentences are mostly the same but the ﬁrst one adds context about an additional dolphin scheme.

Original: a h a r p i s k e y b
Revised: a h a r p i s k e y b

Original: o a r d m a n p l a y i n g a harp is keyboard man playing 0.
Revised: o a r d m a n p l a y i n g a harp is keyboard man playing 0.

Original: (a) Examplea s d o l p
Revised: (a) Examplea s d o l p

Original: h
Revised: h

Original: i n e n g a g e s c
Revised: i n e n g a g e s c

Original: h e m e s e t t
Revised: h e m e s e t t

Original: h e u p w e ll as dolphin engage scheme set the up well 0.
Revised: h e u p w e ll as dolphin engage scheme set the up well 0.

Original: (b) Exampleg o i n g g u e ss i n e x t r e w a n t going guess i next re want 0.
Revised: (b) Exampleg o i n g g u e ss i n e x t r e w a n t going guess i next re want 0.

Original: (c) Example
Revised: (c) Example

Original: In addition to the analogy shown in Figure 1, the other two top analogies from AbE are:
Revised: In addition to the analogy shown in Figure 1, the other two top analogies from AbE are:

Original: The analogy in Figure 1 and the ﬁrst analogy above are good matches because like the input pair, each analogous pair makes the same statement but one of the sentences gives more context (a group in North America and because of last week’s actions, Singapore being a small city-state).
Revised: The analogy in Figure 1 and the ﬁrst analogy above are good matches because like the input pair, each analogous pair makes the same statement but one of the sentences gives more context (a group in North America and because of last week’s actions, Singapore being a small city-state).

Original: The second analogy is more distant (about two different ozone thresholds) but its BB distance is also higher.
Revised: The second analogy is more distant (about two different ozone thresholds) but its BB distance is also higher.

Original: The distance contribution matrix given by FbFull is plotted in Figure 2b.
Revised: The distance contribution matrix given by FbFull is plotted in Figure 2b.

Original: For clarity, only rows/columns with absolute sum greater than 0 .
Revised: For clarity, only rows/columns with absolute sum greater than 0 .

Original: 01 are shown.
Revised: 01 are shown.

Original: Several words with the largest contributions come from the additional phrase about the dolphin scheme.
Revised: Several words with the largest contributions come from the additional phrase about the dolphin scheme.

Original: The substitution of the verb “set up” for “engage” is also highlighted.
Revised: The substitution of the verb “set up” for “engage” is also highlighted.

Original: Example 3:
Revised: Example 3:

Original: The third pair is both more complex than Example 1 and less similar than Example 2:
Revised: The third pair is both more complex than Example 1 and less similar than Example 2:

Original: (a) It depends on what you want to do next, and where you want to do it.
Revised: (a) It depends on what you want to do next, and where you want to do it.

Original: (b) I guess it depends on what you’re going to do.
Revised: (b) I guess it depends on what you’re going to do.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: Figure 2c shows the distance contribution matrix produced by FbFull, again restricted to signiﬁcant rows/columns.
Revised: Figure 2c shows the distance contribution matrix produced by FbFull, again restricted to signiﬁcant rows/columns.

Original: The most important contributions identiﬁed are the substitution of “[a]re going” for “want” and the addition of “I guess” in sentence b).
Revised: The most important contributions identiﬁed are the substitution of “[a]re going” for “want” and the addition of “I guess” in sentence b).

Original: Of minor importance but interesting to note is that the word “next” in sentence a) would have a larger contribution but it is offset by negative contributions from the (“next”, “going”) and (“next”, “guess”) entries.
Revised: Of minor importance but interesting to note is that the word “next” in sentence a) would have a larger contribution but it is offset by negative contributions from the (“next”, “going”) and (“next”, “guess”) entries.

Original: Both “next” and “going” are indicative of future action.
Revised: Both “next” and “going” are indicative of future action.

Original: Below is the top analogy for Example 3:
Revised: Below is the top analogy for Example 3:

Original: (a) I prefer to run the second half 1-2 minutes faster then the ﬁrst.
Revised: (a) I prefer to run the second half 1-2 minutes faster then the ﬁrst.

Original: (b) I would deﬁnitely go for a slightly slower ﬁrst half.
Revised: (b) I would deﬁnitely go for a slightly slower ﬁrst half.

Original: δ BB ( x, y )
Revised: δ BB ( x, y )

Original: = 0 .
Revised: = 0 .

Original: Both sentences express the same idea (second half faster than ﬁrst half) but in different ways, similar to the input pair.
Revised: Both sentences express the same idea (second half faster than ﬁrst half) but in different ways, similar to the input pair.

Original: Two more analogies are discussed in Appendix K.
Revised: Two more analogies are discussed in Appendix K.

Original: 5. U SER S TUDY
Revised: 5. U SER S TUDY

Original: We designed and conducted a human based evaluation to investigate ﬁve local explainability methods.
Revised: We designed and conducted a human based evaluation to investigate ﬁve local explainability methods.

Original: Methods: Besides the proposed FbFull and AbE methods, the three other approaches evaluated in the user study are: feature-based explanation with diagonal A (FbDiag); ProtoDash
Revised: Methods: Besides the proposed FbFull and AbE methods, the three other approaches evaluated in the user study are: feature-based explanation with diagonal A (FbDiag); ProtoDash

Original: (PDash) (Gurumoorthy et al., 2019), a state-of-the-art exemplar explanation method 3 ; and Direction Similarity (DirSim), which ﬁnds analogies like AbE but using only the direction similarity term D ( z i , x ) in equation 4.
Revised: (PDash) (Gurumoorthy et al., 2019), a state-of-the-art exemplar explanation method 3 ; and Direction Similarity (DirSim), which ﬁnds analogies like AbE but using only the direction similarity term D ( z i , x ) in equation 4.

Original: Given the novelty of our setting, we thought these to be reasonable competitors.
Revised: Given the novelty of our setting, we thought these to be reasonable competitors.

Original: Setup:
Revised: Setup:

Original: For each pair of sentences in the STS dataset, users were instructed to use the provided explanations to estimate the similarity of the pair per a black box similarity model.
Revised: For each pair of sentences in the STS dataset, users were instructed to use the provided explanations to estimate the similarity of the pair per a black box similarity model.

Original: As mentioned in
Revised: As mentioned in

Original: Section 5.1, the black box model produces cosine distances in [0 , 1] based on a universal sentence encoder (Cer et al., 2018b).
Revised: Section 5.1, the black box model produces cosine distances in [0 , 1] based on a universal sentence encoder (Cer et al., 2018b).

Original: To be more consumable to humans, the outputs of the black box model were discretized into three categories: Similar ( 0 − 0 .
Revised: To be more consumable to humans, the outputs of the black box model were discretized into three categories: Similar ( 0 − 0 .

Original: 25 distance), Somewhat similar ( 0 .
Revised: 25 distance), Somewhat similar ( 0 .

Original: 25 − 0 .distance) and Dissimilar ( > 0 .
Revised: 25 − 0 .distance) and Dissimilar ( > 0 .

Original: 75 distance).
Revised: 75 distance).

Original: Users were asked to predict one of these categories or “can’t say” if they were unable to do so.
Revised: Users were asked to predict one of these categories or “can’t say” if they were unable to do so.

Original: Screenshots illustrating this are in Appendix M and the full user study is attached in Appendix O. Predicting black box outputs is a standard procedure to measure efﬁcacy of explanations (Ramamurthy et al., 2020; Ribeiro et al., 2016; Lipton, 2016).
Revised: Screenshots illustrating this are in Appendix M and the full user study is attached in Appendix Q. Predicting black box outputs is a standard procedure to measure efﬁcacy of explanations (Ramamurthy et al., 2020; Ribeiro et al., 2016; Lipton, 2016).

Original: In the survey, 10 pairs of sentences were selected randomly in stratiﬁed fashion from the test set such that four were similar, four were somewhat similar, and the remaining two were dissimilar as per the black box.
Revised: In the survey, 10 pairs of sentences were selected randomly in stratiﬁed fashion from the test set such that four were similar, four were somewhat similar, and the remaining two were dissimilar as per the black box.

Original: This was done to be consistent with the distribution of sentence pairs in the dataset with respect to these categories.
Revised: This was done to be consistent with the distribution of sentence pairs in the dataset with respect to these categories.

Original: Also, half the pairs selected were short sentence pairs where the number of words in each sentence was typically ≤ 10 , while for the remaining pairs (i.e. long sentence pairs) the numbers of words were typically closer to 20 .
Revised: Also, half the pairs selected were short sentence pairs where the number of words in each sentence was typically ≤ 10 , while for the remaining pairs (i.e. long sentence pairs) the numbers of words were typically closer to 20 .

Original: This was done to test the explanation methods for different levels of complexity in the input, thus making our conclusions more robust.
Revised: This was done to test the explanation methods for different levels of complexity in the input, thus making our conclusions more robust.

Original: The users were blinded to which explanation method produced a particular explanation.
Revised: The users were blinded to which explanation method produced a particular explanation.

Original: The survey had 30 questions where each question corresponded to an explanation for a sentence pair.
Revised: The survey had 30 questions where each question corresponded to an explanation for a sentence pair.

Original: Given that there were 10 sentence pairs, we randomly chose three methods per pair, which mapped to three different questions.
Revised: Given that there were 10 sentence pairs, we randomly chose three methods per pair, which mapped to three different questions.

Original: For feature-based explanations, the output from the explanation model was provided along with a set of important words, corresponding to rows in the A matrix with the largest sums in absolute value.
Revised: For feature-based explanations, the output from the explanation model was provided along with a set of important words, corresponding to rows in the A matrix with the largest sums in absolute value.

Original: For analogy-based explanations, black box outputs were provided for the analogies only (not for the input sentence pair), selected from the STS dev set.
Revised: For analogy-based explanations, black box outputs were provided for the analogies only (not for the input sentence pair), selected from the STS dev set.

Original: We did this to allow the users to calibrate the black box relative to the explanations, and without which it would be impossible to estimate the similarity of the sentence pair in question.
Revised: We did this to allow the users to calibrate the black box relative to the explanations, and without which it would be impossible to estimate the similarity of the sentence pair in question.

Original: More importantly though, all this information would be available to the user in a real scenario where they are given explanations.
Revised: More importantly though, all this information would be available to the user in a real scenario where they are given explanations.

Original: We leveraged Google Forms for our study.
Revised: We leveraged Google Forms for our study.

Original: 41 participants took it with most of them having backgrounds in data science, engineering and business analytics.
Revised: 41 participants took it with most of them having backgrounds in data science, engineering and business analytics.

Original: We did this as recent work shows that most consumers of such explanations have these backgrounds (Bhatt et al., 2020).
Revised: We did this as recent work shows that most consumers of such explanations have these backgrounds (Bhatt et al., 2020).

Original: Observations:
Revised: Observations:

Original: In Figure 3 we see a summary of the results from our user study.
Revised: In Figure 3 we see a summary of the results from our user study.

Original: In the left ﬁgure (all sentences), we observe that AbE and FbFull signiﬁcantly outperform both exemplar-based and feature-based baselines.
Revised: In the left ﬁgure (all sentences), we observe that AbE and FbFull signiﬁcantly outperform both exemplar-based and feature-based baselines.

Original: AbE seems to be slightly better than FbFull; however, the difference is not statistically signiﬁcant.
Revised: AbE seems to be slightly better than FbFull; however, the difference is not statistically signiﬁcant.

Original: While the results in Section 5.3 show that both of these methods have high ﬁdelity, this was not known to the participants, who instead had to use the provided reasons (analogies or important words) to decide whether to accept the outputs of the explanation methods.
Revised: While the results in Section 5.3 show that both of these methods have high ﬁdelity, this was not known to the participants, who instead had to use the provided reasons (analogies or important words) to decide whether to accept the outputs of the explanation methods.

Original: The good performance of AbE and FbFull suggests that the provided reasons are sensible to users.
Revised: The good performance of AbE and FbFull suggests that the provided reasons are sensible to users.

Original: In the center ﬁgure (short sentences), DirSim is the closest competitor, which suggests that the black box model is outputting distances that accord with intuition.
Revised: In the center ﬁgure (short sentences), DirSim is the closest competitor, which suggests that the black box model is outputting distances that accord with intuition.

Original: FbDiag does worst here, signaling the importance of looking at interactions between words.
Revised: FbDiag does worst here, signaling the importance of looking at interactions between words.

Original: However in the right ﬁgure (long sentences),
Revised: However in the right ﬁgure (long sentences),

Original: [Figure]
Revised: [Figure]

Original: FbDiag is the closest competitor and DirSim is the worst, hinting that predicting the black box similarity becomes harder based on intuition and certain key words are important to focus on independent of everything else.
Revised: FbDiag is the closest competitor and DirSim is the worst, hinting that predicting the black box similarity becomes harder based on intuition and certain key words are important to focus on independent of everything else.

Original: We also solicited (optional) user feedback (provided in Appendix N).
Revised: We also solicited (optional) user feedback (provided in Appendix N).

Original: From the comments, it appeared that there were two main groups.
Revised: From the comments, it appeared that there were two main groups.

Original: One preferred analogies as they felt they gave them more information to make the decision.
Revised: One preferred analogies as they felt they gave them more information to make the decision.

Original: This is seen from comments such as “The examples [analogies] seem to be more reliable than the verbal reason [words].”
Revised: This is seen from comments such as “The examples [analogies] seem to be more reliable than the verbal reason [words].”

Original: There was support for having multiple diverse analogies to increase conﬁdence in a prediction, as seen in “The range of examples may be useful, as some questions have all three examples in the same class.”
Revised: There was support for having multiple diverse analogies to increase conﬁdence in a prediction, as seen in “The range of examples may be useful, as some questions have all three examples in the same class.”

Original: While one would expect this beneﬁt to diminish without diversity in the multiple analogies, this aspect was not explicitly tested.
Revised: While one would expect this beneﬁt to diminish without diversity in the multiple analogies, this aspect was not explicitly tested.

Original: The second group felt the feature-based explanations were better given their precision and brevity.
Revised: The second group felt the feature-based explanations were better given their precision and brevity.

Original: An example comment here was “I ﬁnd the explanation with the difference between the sentences easier to reason about.”
Revised: An example comment here was “I ﬁnd the explanation with the difference between the sentences easier to reason about.”

Original: A couple of people also said that providing both the feature-based and analogy-based explanations would be useful as they somewhat complement each other and can help cross-verify one’s assessment.
Revised: A couple of people also said that providing both the feature-based and analogy-based explanations would be useful as they somewhat complement each other and can help cross-verify one’s assessment.

Original: 5. Q UANTITATIVE E XPERIMENTS
Revised: 5. Q UANTITATIVE E XPERIMENTS

Original: This section presents evaluations of the ﬁdelity of various explanation methods with respect to the black box similarity model’s outputs.
Revised: This section presents evaluations of the ﬁdelity of various explanation methods with respect to the black box similarity model’s outputs.

Original: Methods: In addition to the ﬁve local explanation methods considered in Section 5.2, we evaluate a globally interpretable model, global feature-based full-matrix explanations (GFbFull), LIME (Ribeiro et al., 2016), and Joint Search LIME (JSLIME) (Hamilton et al., 2021).
Revised: Methods: In addition to the ﬁve local explanation methods considered in Section 5.2, we evaluate a globally interpretable model, global feature-based full-matrix explanations (GFbFull), LIME (Ribeiro et al., 2016), and Joint Search LIME (JSLIME) (Hamilton et al., 2021).

Original: GFbFull uses a Mahalanobis model like in Section 4.1 but ﬁt on the entire dataset instead of a perturbation neighborhood N xy .
Revised: GFbFull uses a Mahalanobis model like in Section 4.1 but ﬁt on the entire dataset instead of a perturbation neighborhood N xy .

Original: To run GFbFull on the STS dataset, we chose only the top 500 words in the test set vocabulary according to tf-idf scores to limit the computational complexity.
Revised: To run GFbFull on the STS dataset, we chose only the top 500 words in the test set vocabulary according to tf-idf scores to limit the computational complexity.

Original: For all methods, explanations were generated using the test set and for AbE, DirSim, and PDash, we use the validation set to select the analogies.
Revised: For all methods, explanations were generated using the test set and for AbE, DirSim, and PDash, we use the validation set to select the analogies.

Original: Data and Black Box Models:
Revised: Data and Black Box Models:

Original: In addition to the STS dataset, we use two other datasets along with attendant black box models as described below: UCI Iris (Dheeru & Karra Taniskidou, 2017) and
Revised: In addition to the STS dataset, we use two other datasets along with attendant black box models as described below: UCI Iris (Dheeru & Karra Taniskidou, 2017) and

Original: Medical Expenditure Panel Survey (MEPS) (MEP).
Revised: Medical Expenditure Panel Survey (MEPS) (MEP).

Original: The supplement has more details on datasets, black box models, and neighborhoods for feature-based explanations.
Revised: The supplement has more details on datasets, black box models, and neighborhoods for feature-based explanations.

Original: For Iris and MEPS, 5 -fold cross-validation was performed.
Revised: For Iris and MEPS, 5 -fold cross-validation was performed.

Original: For Iris, pairs of examples were exhaustively enumerated and labeled as similar or dissimilar based on the species labels.
Revised: For Iris, pairs of examples were exhaustively enumerated and labeled as similar or dissimilar based on the species labels.

Original: A Siamese network was trained on these labeled pairs as the black box model δ BB , achieving a mean absolute error (MAE) of 0 .
Revised: A Siamese network was trained on these labeled pairs as the black box model δ BB , achieving a mean absolute error (MAE) of 0 .

Original: 400 ± 0 .
Revised: 400 ± 0 .

Original: 044 with respect to the ground truth distances and Pearson’s r of 0 .
Revised: 044 with respect to the ground truth distances and Pearson’s r of 0 .

Original: 370 ± 0 .
Revised: 370 ± 0 .

Original: For MEPS, we found that tree-based models worked better for this largely categorical dataset.
Revised: For MEPS, we found that tree-based models worked better for this largely categorical dataset.

Original: Accordingly, we ﬁrst trained a Random Forest regressor to predict healthcare utilization, achieving a test R 2 value of 0 .
Revised: Accordingly, we ﬁrst trained a Random Forest regressor to predict healthcare utilization, achieving a test R 2 value of 0 .

Original: 381 ± 0 . 017 .
Revised: 381 ± 0 . 017 .

Original: The black box function δ BB ( x, y ) was then obtained as the distance between the leaf embeddings (Zhu et al., 2016) of x , y from the random forest.
Revised: The black box function δ BB ( x, y ) was then obtained as the distance between the leaf embeddings (Zhu et al., 2016) of x , y from the random forest.

Original: Note therefore that δ BB ( x, y ) is a distance function of two inputs, not a regression function of one input.
Revised: Note therefore that δ BB ( x, y ) is a distance function of two inputs, not a regression function of one input.

Original: Pairs of examples to be explained were generated by stratiﬁed random sampling based on δ BB ( ., . ) values.
Revised: Pairs of examples to be explained were generated by stratiﬁed random sampling based on δ BB ( ., . ) values.

Original: For feature-based explanations we chose 10000 pairs each from the validation and test set of each fold.
Revised: For feature-based explanations we chose 10000 pairs each from the validation and test set of each fold.

Original: [Table]
Revised: [Table]

Original: For AbE, DirSim, and PDash, we chose 1000 pairs to limit the computational complexity.
Revised: For AbE, DirSim, and PDash, we chose 1000 pairs to limit the computational complexity.

Original: For AbE, we used λ 1 = 1 . 0 and λ 2 = 0 .
Revised: For AbE, we used λ 1 = 1 . 0 and λ 2 = 0 .

Original: 01 for both MEPS and Iris.
Revised: 01 for both MEPS and Iris.

Original: For feature-based explanations, we present comparisons of generalized inﬁdelity (Ramamurthy et al., 2020).
Revised: For feature-based explanations, we present comparisons of generalized inﬁdelity (Ramamurthy et al., 2020).

Original: This tests generalization by computing the MAE between the black-box distance for an input instance pair and the explanation of the closest neighboring test instance pair.
Revised: This tests generalization by computing the MAE between the black-box distance for an input instance pair and the explanation of the closest neighboring test instance pair.

Original: Table 1 shows the generalized inﬁdelity for FbFull, FbDiag, LIME, and JSLIME with respect to the black box predictions.
Revised: Table 1 shows the generalized inﬁdelity for FbFull, FbDiag, LIME, and JSLIME with respect to the black box predictions.

Original: Since GFbFull computes global explanations, we cannot obtain this measure.
Revised: Since GFbFull computes global explanations, we cannot obtain this measure.

Original: Generalized ﬁdelity computed using Pearson’s r and non-generalized MAE/Pearson’s r for all methods including
Revised: Generalized ﬁdelity computed using Pearson’s r and non-generalized MAE/Pearson’s r for all methods including

Original: GFbFull are presented in Appendix I. Appendix H also presents more descriptions of metrics used.
Revised: GFbFull are presented in Appendix I. Appendix H also presents more descriptions of metrics used.

Original: From Table 1, FbFull/FbDiag have superior performance.
Revised: From Table 1, FbFull/FbDiag have superior performance.

Original: This suggests that they provide better generalization in the neighborhood by virtue of Mahalanobis distance being a metric.
Revised: This suggests that they provide better generalization in the neighborhood by virtue of Mahalanobis distance being a metric.

Original: We do not expect LIME to perform well as discussed in Section 4.1, but JSLIME also has poor performance with this metric since it likely overﬁts because of the lack of constraints on A .
Revised: We do not expect LIME to perform well as discussed in Section 4.1, but JSLIME also has poor performance with this metric since it likely overﬁts because of the lack of constraints on A .

Original: Since all the black box predictions are between 0 and 1, it is possible to compare these three datasets.
Revised: Since all the black box predictions are between 0 and 1, it is possible to compare these three datasets.

Original: The methods seem to perform best with MEPS, followed by STS and Iris.
Revised: The methods seem to perform best with MEPS, followed by STS and Iris.

Original: The MEPS dataset, even though the largest of the three has two advantages.
Revised: The MEPS dataset, even though the largest of the three has two advantages.

Original: The variables are binary (dummy coded from categorical) which possibly leads to better ﬁtting explanations, and the search space for computing the generalized metric is large, which means that the likelihood of ﬁnding a neighboring test instance pair with a good explanation is high.
Revised: The variables are binary (dummy coded from categorical) which possibly leads to better ﬁtting explanations, and the search space for computing the generalized metric is large, which means that the likelihood of ﬁnding a neighboring test instance pair with a good explanation is high.

Original: For STS, the black box universal sentence encoder seems to agree with the local Mahalanobis distance approximation and to some extent even with the diagonal approximation.
Revised: For STS, the black box universal sentence encoder seems to agree with the local Mahalanobis distance approximation and to some extent even with the diagonal approximation.

Original: Iris has the worst performance possibly because the dataset is so small that a Siamese neural network cannot approximate the underlying similarity function well, and also because the search space for computing the generalized metric is quite small.
Revised: Iris has the worst performance possibly because the dataset is so small that a Siamese neural network cannot approximate the underlying similarity function well, and also because the search space for computing the generalized metric is quite small.

Original: The inﬁdelity (MAE) of the analogy explanation methods (AbE, DirSim, and PDash) is illustrated in Figure 4.
Revised: The inﬁdelity (MAE) of the analogy explanation methods (AbE, DirSim, and PDash) is illustrated in Figure 4.

Original: Given a set of analogies z 1 , . . .
Revised: Given a set of analogies z 1 , . . .

Original: , z k , the prediction of the explainer is computed as the average of the black box predictions δ BB ( z 1 ) , . . .
Revised: , z k , the prediction of the explainer is computed as the average of the black box predictions δ BB ( z 1 ) , . . .

Original: , δ BB ( z k ) for the analogies.
Revised: , δ BB ( z k ) for the analogies.

Original: The proposed AbE method dominates the other two baselines because of the explicit inclusion of the black box ﬁdelity term in the objective.
Revised: The proposed AbE method dominates the other two baselines because of the explicit inclusion of the black box ﬁdelity term in the objective.

Original: For Iris and STS, the MAE of AbE steadily increases with the number of analogies.
Revised: For Iris and STS, the MAE of AbE steadily increases with the number of analogies.

Original: This is expected because of the trade-off between the ﬁdelity term and the diversity term in (equation 2) as k increases.
Revised: This is expected because of the trade-off between the ﬁdelity term and the diversity term in (equation 2) as k increases.

Original: For MEPS, the MAE of AbE very slowly reduces and ﬂattens out.
Revised: For MEPS, the MAE of AbE very slowly reduces and ﬂattens out.

Original: This could be due to the greater availability of high-ﬁdelity analogous pairs in MEPS.
Revised: This could be due to the greater availability of high-ﬁdelity analogous pairs in MEPS.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In summary, we have provided (model agnostic) local explanations for similarity learners, in both the more familiar form of feature attributions as well as the more novel form of analogies.
Revised: In summary, we have provided (model agnostic) local explanations for similarity learners, in both the more familiar form of feature attributions as well as the more novel form of analogies.

Original: The experimental results indicate that the resulting explanations have high ﬁdelity, appear useful to humans in judging the black box’s behavior, and offer qualitative insights.
Revised: The experimental results indicate that the resulting explanations have high ﬁdelity, appear useful to humans in judging the black box’s behavior, and offer qualitative insights.

Original: For the analogy-based method, the selection of analogies is signiﬁcantly inﬂuenced by the analogy closeness term G ( z i , x ) in (equation 2).
Revised: For the analogy-based method, the selection of analogies is signiﬁcantly inﬂuenced by the analogy closeness term G ( z i , x ) in (equation 2).

Original: Herein we have used direction similarity equation 4, which is convenient to compute given an embedding and appears to capture word and phrasing relations well in the STS dataset.
Revised: Herein we have used direction similarity equation 4, which is convenient to compute given an embedding and appears to capture word and phrasing relations well in the STS dataset.

Original: It would be interesting to devise more sophisticated analogy closeness functions, tailored to the notion of analogy in a given context.
Revised: It would be interesting to devise more sophisticated analogy closeness functions, tailored to the notion of analogy in a given context.

Original: It is also of interest to extend this work from explaining pairwise relationships to tasks such as ranking.
Revised: It is also of interest to extend this work from explaining pairwise relationships to tasks such as ranking.

Original: We thus hope that the approaches developed here could become meta-approaches for handling multiple types of relationships.
Revised: We thus hope that the approaches developed here could become meta-approaches for handling multiple types of relationships.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Explanation methods can be hugely beneﬁcial to stakeholders and can provide real power to end users to ﬁght unfair automated decisions.
Revised: Explanation methods can be hugely beneﬁcial to stakeholders and can provide real power to end users to ﬁght unfair automated decisions.

Original: However, they are also prone to risks.
Revised: However, they are also prone to risks.

Original: The risks of using post-hoc explanations for similarity learning are similar to those of using post-hoc explanations for classiﬁcation/regression.
Revised: The risks of using post-hoc explanations for similarity learning are similar to those of using post-hoc explanations for classiﬁcation/regression.

Original: Firstly, explanations may not be completely faithful to the black box and hence misleading.
Revised: Firstly, explanations may not be completely faithful to the black box and hence misleading.

Original: Secondly, there are be many conﬂicting explanations that can have high ﬁdelity with respect to the black box predictions.
Revised: Secondly, there are be many conﬂicting explanations that can have high ﬁdelity with respect to the black box predictions.

Original: This is why we think placing reasonable constraints (such as forcing the explainer to approximate a distance metric) is essential.
Revised: This is why we think placing reasonable constraints (such as forcing the explainer to approximate a distance metric) is essential.

Original: Thirdly, explanations may pose privacy/security risks and reveal the inner workings of the black box.
Revised: Thirdly, explanations may pose privacy/security risks and reveal the inner workings of the black box.

Original: With careful understanding of these issues, many of these risks can be controlled and mitigated.
Revised: With careful understanding of these issues, many of these risks can be controlled and mitigated.

Original: R EPRODUCIBILITY D ISCUSSION
Revised: R EPRODUCIBILITY D ISCUSSION

Original: Experimental details and results are provided in Section 5 of the main paper.
Revised: Experimental details and results are provided in Section 5 of the main paper.

Original: All datasets are public and their details are provided in Appendix F. Hyperparameters used are discussed in Appendix G.
Revised: All datasets are public and their details are provided in Appendix F. Hyperparameters used are discussed in Appendix G.

Original: The metrics used in evaluations are described in H. The complete user study questions are provided in Section O. Code will be provided during the discussion phase through an anonymized link.
Revised: The metrics used in evaluations are described in H. The complete user study questions are provided in Section Q. Code will be provided during the discussion phase through an anonymized link.

Original: A PPENDIX A O THER E XPLAINABILITY M ETHODS
Revised: A PPENDIX A O THER E XPLAINABILITY M ETHODS

Original: A large body of work on XAI can be said to belong to either local explanations (Ribeiro et al., 2016;
Revised: A large body of work on XAI can be said to belong to either local explanations (Ribeiro et al., 2016;

Original: Lundberg & Lee, 2017; Gurumoorthy et al., 2019), global explanations (Hinton et al., 2015; Bastani et al., 2017; Bucilu ˇ a et al., 2006; Dhurandhar et al., 2018b; 2020), directly interpretable models (Caruana et al., 2015; Rudin, 2018;
Revised: Lundberg & Lee, 2017; Gurumoorthy et al., 2019), global explanations (Hinton et al., 2015; Bastani et al., 2017; Bucilu ˇ a et al., 2006; Dhurandhar et al., 2018b; 2020), directly interpretable models (Caruana et al., 2015; Rudin, 2018;

Original: Su et al., 2016;
Revised: Su et al., 2016;

Original: Dash et al., 2018; Sipser, 2013) or visualizationbased methods (Hendricks et al., 2016).
Revised: Dash et al., 2018; Sipser, 2013) or visualizationbased methods (Hendricks et al., 2016).

Original: Among these categories, local explainability methods are the most relevant to our current endeavor.
Revised: Among these categories, local explainability methods are the most relevant to our current endeavor.

Original: Local explanation methods generate explanations per example for a given black box.
Revised: Local explanation methods generate explanations per example for a given black box.

Original: Methods in this category are either feature-based (Ribeiro et al., 2016; Lundberg & Lee, 2017; Bach et al., 2015; Dhurandhar et al., 2018a; Mothilal et al., 2019) or exemplar-based (Gurumoorthy et al., 2019;
Revised: Methods in this category are either feature-based (Ribeiro et al., 2016; Lundberg & Lee, 2017; Bach et al., 2015; Dhurandhar et al., 2018a; Mothilal et al., 2019) or exemplar-based (Gurumoorthy et al., 2019;

Original: Kim et al., 2016).
Revised: Kim et al., 2016).

Original: There are also a number of methods in this category speciﬁcally designed for images (Simonyan et al., 2013; Bach et al., 2015; Guidotti et al., 2019; Lampridis et al., 2020; Selvaraju et al., 2020).
Revised: There are also a number of methods in this category speciﬁcally designed for images (Simonyan et al., 2013; Bach et al., 2015; Guidotti et al., 2019; Lampridis et al., 2020; Selvaraju et al., 2020).

Original: However, all of the above methods are predominantly applicable to the classiﬁcation setting and in a smaller number of cases to regression.
Revised: However, all of the above methods are predominantly applicable to the classiﬁcation setting and in a smaller number of cases to regression.

Original: Global explainability methods try to build an interpretable model on the entire dataset using information from the black-box model with the intention of approaching the black-box models performance.
Revised: Global explainability methods try to build an interpretable model on the entire dataset using information from the black-box model with the intention of approaching the black-box models performance.

Original: Methods in this category either use predictions (soft or hard) of the black-box model to train simpler interpretable models Hinton et al.
Revised: Methods in this category either use predictions (soft or hard) of the black-box model to train simpler interpretable models Hinton et al.

Original: ; Bastani et al.
Revised: ; Bastani et al.

Original: ; Buciluˇa et al.
Revised: ; Buciluˇa et al.

Original: or extract weights based on the prediction conﬁdences reweighting the dataset Dhurandhar et al.
Revised: or extract weights based on the prediction conﬁdences reweighting the dataset Dhurandhar et al.

Original: (2018b; 2020).
Revised: (2018b; 2020).

Original: Directly interpretable methods include some of the traditional models such as decision trees or logistic regression.
Revised: Directly interpretable methods include some of the traditional models such as decision trees or logistic regression.

Original: There has been a lot of effort recently to efﬁciently and accurately learn rule lists Rudin (2018) or two-level boolean rules Su et al.
Revised: There has been a lot of effort recently to efﬁciently and accurately learn rule lists Rudin (2018) or two-level boolean rules Su et al.

Original: (2016) or decision sets Sipser (2013).
Revised: (2016) or decision sets Sipser (2013).

Original: There has also been work inspired by other ﬁelds such as psychometrics Idé & Dhurandhar (2017) and healthcare Caruana et al.
Revised: There has also been work inspired by other ﬁelds such as psychometrics Idé & Dhurandhar (2017) and healthcare Caruana et al.

Original: Visualization based methods try to visualize the inner neurons or set of neurons in a layer of a neural network Hendricks et al.
Revised: Visualization based methods try to visualize the inner neurons or set of neurons in a layer of a neural network Hendricks et al.

Original: The idea is that by exposing such representations one may be able to gauge if the neural network is in fact capturing semantically meaningful high level features.
Revised: The idea is that by exposing such representations one may be able to gauge if the neural network is in fact capturing semantically meaningful high level features.

Original: B P ROOF OF S UBMODULARITY (L EMMA 4.1)
Revised: B P ROOF OF S UBMODULARITY (L EMMA 4.1)

Original: Proof.
Revised: Proof.

Original: Consider two sets S and T consisting of elements z (i.e. analogous pairs) as deﬁned before, where S ⊆ T .
Revised: Consider two sets S and T consisting of elements z (i.e. analogous pairs) as deﬁned before, where S ⊆ T .

Original: Let w be a pair (cid:54)∈ T and x be an input pair that we want to explain.
Revised: Let w be a pair (cid:54)∈ T and x be an input pair that we want to explain.

Original: Then for any valid
Revised: Then for any valid

Original: S and T , we have
Revised: S and T , we have

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Subtracting equation equation 6 from equation 5 and ignoring λ 2 as it just scales the difference without changing the sign gives us
Revised: Subtracting equation equation 6 from equation 5 and ignoring λ 2 as it just scales the difference without changing the sign gives us

Original: [Equation]
Revised: [Equation]

Original: Thus, the function f ( . ) has diminishing returns property.
Revised: Thus, the function f ( . ) has diminishing returns property.

Original: C G REEDY A PPROXIMATE A LGORITHM FOR S OLVING EQUATION
Revised: C G REEDY A PPROXIMATE A LGORITHM FOR S OLVING EQUATION

Original: The only reliable software that was available for solving the submodular minimization was the SFO
Revised: The only reliable software that was available for solving the submodular minimization was the SFO

Original: MATLAB package (Krause, 2010).
Revised: MATLAB package (Krause, 2010).

Original: However, we faced the following challenges - (a) it was quite slow to run the exact optimization and since we had to compute thousands of local explanations, it would have taken an unreasonably long time, (b) we wanted k sparse solutions (not unconstrained outputs)
Revised: However, we faced the following challenges - (a) it was quite slow to run the exact optimization and since we had to compute thousands of local explanations, it would have taken an unreasonably long time, (b) we wanted k sparse solutions (not unconstrained outputs)

Original: (c) the optimization was quite sensitive in the exact setting to the hyperparameters λ 1 and λ 2 , (c) attempts to speed up the execution by parallelizing it would require algorithmic innovations, (d) MATLAB needed paid licenses.
Revised: (c) the optimization was quite sensitive in the exact setting to the hyperparameters λ 1 and λ 2 , (c) attempts to speed up the execution by parallelizing it would require algorithmic innovations, (d) MATLAB needed paid licenses.

Original: Hence for the purposes of having better control, speed, and efﬁciency, we implemented a greedy approximate version of the objective in (equation 2).
Revised: Hence for the purposes of having better control, speed, and efﬁciency, we implemented a greedy approximate version of the objective in (equation 2).

Original: The greedy approach chooses one analogous pair ( z i ) to minimize the current objective value and keeps repeating it until k pairs are chosen.
Revised: The greedy approach chooses one analogous pair ( z i ) to minimize the current objective value and keeps repeating it until k pairs are chosen.

Revised: The greedy algorithm is provided in Algorithm 2.

Original: D C OMPUTATIONAL C OMPLEXITIES
Revised: D C OMPUTATIONAL C OMPLEXITIES

Original: The FbFull method involves solving an SDP which has at least a time complexity of O ( d 3 ) where d is the number of features since each iteration usually involves solving a linear system or inverting a matrix of that size.
Revised: The FbFull method involves solving an SDP which has at least a time complexity of O ( d 3 ) where d is the number of features since each iteration usually involves solving a linear system or inverting a matrix of that size.

Original: However, it is not apparent how the CVXPY package we use sets up and solves this problem, which could alter this complexity.
Revised: However, it is not apparent how the CVXPY package we use sets up and solves this problem, which could alter this complexity.

Original: We implement a non-negative sparse minimization for FbDiag with k non-zeros and for this case the computational complexity if O ( Nk 2 ) where N is the number of perturbations used.
Revised: We implement a non-negative sparse minimization for FbDiag with k non-zeros and for this case the computational complexity if O ( Nk 2 ) where N is the number of perturbations used.

Original: For the proposed AbE method, the objective function is submodular the k-sparse minimization algorithm has an approximation guarantee of (cid:113) k ln k and runs in O ( k 4 . 5 ) time.
Revised: For the proposed AbE method, the objective function is submodular the k-sparse minimization algorithm has an approximation guarantee of (cid:113) k ln k and runs in O ( k 4 . 5 ) time.

Original: Typically, k is small.
Revised: Typically, k is small.

Original: However, there is no software implementation available for this method to the best of our knowledge.
Revised: However, there is no software implementation available for this method to the best of our knowledge.

Original: Hence we use the greedy method proposed above which has a time complexity of O ( Nk ) , where N is the dataset size.
Revised: Hence we use the greedy method proposed above which has a time complexity of O ( Nk ) , where N is the dataset size.

Original: E A DDITIONAL R EMARKS ON M ETHODS E. J OINT S EARCH LIME
Revised: E A DDITIONAL R EMARKS ON M ETHODS E. J OINT S EARCH LIME

Original: Joint Search LIME (JSLIME) (Hamilton et al., 2021) is a bilinear model akin to ¯ x T
Revised: Joint Search LIME (JSLIME) (Hamilton et al., 2021) is a bilinear model akin to ¯ x T

Original: A ¯ y in our notation, where A is unconstrained and may not even be square, as opposed to FbFull/FbDiag which uses a Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) , where is A semideﬁnite and necessarily square.
Revised: A ¯ y in our notation, where A is unconstrained and may not even be square, as opposed to FbFull/FbDiag which uses a Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) , where is A semideﬁnite and necessarily square.

Original: Both
Revised: Both

Original: FbFull/FbDiag and JSLIME approaches have their merits.
Revised: FbFull/FbDiag and JSLIME approaches have their merits.

Original: Mahalanobis distance is a metric and interpretations can exploit this by decomposing the Mahalanobis distance into distance contributions due to differences in individual features.
Revised: Mahalanobis distance is a metric and interpretations can exploit this by decomposing the Mahalanobis distance into distance contributions due to differences in individual features.

Original: On the other hand, JSLIME can be more ﬂexible because of unconstrained A .
Revised: On the other hand, JSLIME can be more ﬂexible because of unconstrained A .

Original: Hamilton et al.
Revised: Hamilton et al.

Original: (2021) show that it can be used to identify correspondences between parts of two inputs, speciﬁcally a region of a retrieved image that corresponds to a given region of a query image.
Revised: (2021) show that it can be used to identify correspondences between parts of two inputs, speciﬁcally a region of a retrieved image that corresponds to a given region of a query image.

Original: This is a different task from explaining a predicted distance by a decomposition.
Revised: This is a different task from explaining a predicted distance by a decomposition.

Original: Also for tabular data, it is not clear how meaningful the correspondences from JSLIME will be.
Revised: Also for tabular data, it is not clear how meaningful the correspondences from JSLIME will be.

Original: E. E XPLAINING S IMILARITIES V ERSUS D ISTANCES
Revised: E. E XPLAINING S IMILARITIES V ERSUS D ISTANCES

Original: The feature based explanation methods (FbFull and FbDiag) explain the distance between the points x and y given by δ BB ( x, y ) using the Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) .
Revised: The feature based explanation methods (FbFull and FbDiag) explain the distance between the points x and y given by δ BB ( x, y ) using the Mahalanobis distance (¯ x − ¯ y ) T A (¯ x − ¯ y ) .

Original: To understand how this is equivalent to explaining similarities, consider without loss of generality that the maximum value of δ BB ( x, y ) is 1 for any x and y .
Revised: To understand how this is equivalent to explaining similarities, consider without loss of generality that the maximum value of δ BB ( x, y ) is 1 for any x and y .

Original: Now, the explanation model for the simple similarity function 1 − δ BB ( x, y ) is (cid:80) dj
Revised: Now, the explanation model for the simple similarity function 1 − δ BB ( x, y ) is (cid:80) dj

Original: =1 (cid:80) dk
Revised: =1 (cid:80) dk

Original: =1 (1 /d 2 − C jk ) where C jk =
Revised: =1 (1 /d 2 − C jk ) where C jk =

Original: (ˆ x j − ˆ y j )
Revised: (ˆ x j − ˆ y j )

Original: A jk (ˆ x k − ˆ y k ) is the distance contribution discussed in Example 1 (Section 5.1).
Revised: A jk (ˆ x k − ˆ y k ) is the distance contribution discussed in Example 1 (Section 5.1).

Original: Clearly, a low distance contribution C jk results in a high similarity contribution 1 /d 2 − C jk and vice versa.
Revised: Clearly, a low distance contribution C jk results in a high similarity contribution 1 /d 2 − C jk and vice versa.

Original: F D ATA AND B LACK -B OX M ODELS
Revised: F D ATA AND B LACK -B OX M ODELS

Original: For the Iris dataset we created 5 folds of the data with 20% non-overlapping test set in each fold, and the rest of the data in each fold is divided into 80% training and 20% validation samples.
Revised: For the Iris dataset we created 5 folds of the data with 20% non-overlapping test set in each fold, and the rest of the data in each fold is divided into 80% training and 20% validation samples.

Original: For each partition, we create similar and dissimilar pairs exhaustively based on the agreement or disagreement of labels.
Revised: For each partition, we create similar and dissimilar pairs exhaustively based on the agreement or disagreement of labels.

Original: This resulted in an average of 4560 training pairs, 276 validation pairs, and 435 testing pairs per fold.
Revised: This resulted in an average of 4560 training pairs, 276 validation pairs, and 435 testing pairs per fold.

Original: The black-box model used in this case is a paired (conjoined) neural network where each candidate network in the pair has a single dense layer whose parameters are tied to the other candidate and is trained using contrastive loss.
Revised: The black-box model used in this case is a paired (conjoined) neural network where each candidate network in the pair has a single dense layer whose parameters are tied to the other candidate and is trained using contrastive loss.

Original: The mean absolute error (MAE) between the black box predictions, δ BB ( ., . ) , and the ground truth distances between the pairs was 0 .
Revised: The mean absolute error (MAE) between the black box predictions, δ BB ( ., . ) , and the ground truth distances between the pairs was 0 .

Original: 400 ± 0 . 044 , the Pearson’s r was 0 .
Revised: 400 ± 0 . 044 , the Pearson’s r was 0 .

Original: 370 ± 0 .
Revised: 370 ± 0 .

Original: For GFbFull, we chose only the top 500 words in the test set vocabulary according to tf-idf scores to limit the computational complexity.
Revised: For GFbFull, we chose only the top 500 words in the test set vocabulary according to tf-idf scores to limit the computational complexity.

Original: The Medical Expenditure Panel Survey (MEPS) dataset is produced by the US Department of Health and Human Services.
Revised: The Medical Expenditure Panel Survey (MEPS) dataset is produced by the US Department of Health and Human Services.

Original: It is a collection of surveys of families of individuals, medical providers, and employers across the country.
Revised: It is a collection of surveys of families of individuals, medical providers, and employers across the country.

Original: We choose Panel 19 of the survey which consists of a cohort that started in 2014 and consisted of data collected over 5 rounds of interviews over 2014 − 2015 .
Revised: We choose Panel 19 of the survey which consists of a cohort that started in 2014 and consisted of data collected over 5 rounds of interviews over 2014 − 2015 .

Original: The outcome variable was a composite utilization feature that quantiﬁed the total number of healthcare visits of a patient.
Revised: The outcome variable was a composite utilization feature that quantiﬁed the total number of healthcare visits of a patient.

Original: The features used included demographic features, perceived health status, various diagnosis, limitations, and socioeconomic factors.
Revised: The features used included demographic features, perceived health status, various diagnosis, limitations, and socioeconomic factors.

Original: We ﬁlter out records that had a utilization (outcome) of 0, and log-transformed the outcome for modeling.
Revised: We ﬁlter out records that had a utilization (outcome) of 0, and log-transformed the outcome for modeling.

Original: These pre-processing steps resulted in a dataset with 11136 examples and 32 categorical features.
Revised: These pre-processing steps resulted in a dataset with 11136 examples and 32 categorical features.

Original: We used 5 − fold CV using the same approach as in Iris data.
Revised: We used 5 − fold CV using the same approach as in Iris data.

Original: When selecting pairs of examples for explanations, we performed stratiﬁed random sampling based on δ BB ( ., . ) .
Revised: When selecting pairs of examples for explanations, we performed stratiﬁed random sampling based on δ BB ( ., . ) .

Original: For FbFull, FbDiag, and GFbFull, we chose 10000 pairs each from validation and test set for each fold.
Revised: For FbFull, FbDiag, and GFbFull, we chose 10000 pairs each from validation and test set for each fold.

Original: For AbE, DirSim, and PDash, we chose 1000 pairs to limit the computational complexity.
Revised: For AbE, DirSim, and PDash, we chose 1000 pairs to limit the computational complexity.

Original: The regression black-box model used for predicting the utilization outcome was a Random Forest with 500 trees and 50 leaf nodes per tree.
Revised: The regression black-box model used for predicting the utilization outcome was a Random Forest with 500 trees and 50 leaf nodes per tree.

Original: The function δ BB ( x, y ) for a pair ( x, y ) was obtained as the distance between the leaf embeddings Zhu et al.
Revised: The function δ BB ( x, y ) for a pair ( x, y ) was obtained as the distance between the leaf embeddings Zhu et al.

Original: (2016) from the random forests.
Revised: (2016) from the random forests.

Original: The Rperformance measure of the regressor in the test set was 0 .
Revised: The Rperformance measure of the regressor in the test set was 0 .

Original: 381 ± 0 . 017 .
Revised: 381 ± 0 . 017 .

Original: The STS benchmark dataset comprises of 8628 sentence pairs of which 5749 correspond to the training partition, 1500 to the validation partition, and 1379 to the test partition.
Revised: The STS benchmark dataset comprises of 8628 sentence pairs of which 5749 correspond to the training partition, 1500 to the validation partition, and 1379 to the test partition.

Original: Each pair has a ground truth semantic similarity score between 0 (no meaning overlap) and 5 (meaning equivalence).
Revised: Each pair has a ground truth semantic similarity score between 0 (no meaning overlap) and 5 (meaning equivalence).

Original: This can be re-interpreted into a distance measure by subtracting it from 5 and dividing the result by 5 .
Revised: This can be re-interpreted into a distance measure by subtracting it from 5 and dividing the result by 5 .

Original: The black box model used here was the universal sentence encoder 4 (Cer et al., 2018a), which creates a 512 dimensional embedding for each sentence.
Revised: The black box model used here was the universal sentence encoder 4 (Cer et al., 2018a), which creates a 512 dimensional embedding for each sentence.

Original: δ BB ( x, y ) is the cosine distance between the embeddings of x and y .
Revised: δ BB ( x, y ) is the cosine distance between the embeddings of x and y .

Original: The Pearson’s r performance measure of these black-box predictions with respect to the distance between the sentences is 0 .
Revised: The Pearson’s r performance measure of these black-box predictions with respect to the distance between the sentences is 0 .

Original: 787 and the mean absolute error is 0 .
Revised: 787 and the mean absolute error is 0 .

Original: G H YPERPARAMETERS
Revised: G H YPERPARAMETERS

Original: In all datasets, FbFull and GFbFull were computed along with a very small (cid:96) 1 penalty on A ( 10 − 4 (cid:107) A (cid:107) 1 ) added to the objective function (equation 1).
Revised: In all datasets, FbFull and GFbFull were computed along with a very small (cid:96) 1 penalty on A ( 10 − 4 (cid:107) A (cid:107) 1 ) added to the objective function (equation 1).

Original: For FbDiag, we request a maximum of 4 non-zero coefﬁcients for Iris, 10 non-zero coefﬁcients for MEPS, and 5 non-zero coefﬁcients https for STS.
Revised: For FbDiag, we request a maximum of 4 non-zero coefﬁcients for Iris, 10 non-zero coefﬁcients for MEPS, and 5 non-zero coefﬁcients https for STS.

Original: As discussed in Section 5, we set the hyperparameters for analogy-based explanations to a reasonable values as well and do not perform extensive hyperparameter search.
Revised: As discussed in Section 5, we set the hyperparameters for analogy-based explanations to a reasonable values as well and do not perform extensive hyperparameter search.

Original: Perturbations for Local Explanations (FbDiag, FbFull): The input instances x , y are individually perturbed to get the data points ( x i , y i
Revised: Perturbations for Local Explanations (FbDiag, FbFull): The input instances x , y are individually perturbed to get the data points ( x i , y i

Original: ) ∈ N xy
Revised: ) ∈ N xy

Original: (see equation 1).
Revised: (see equation 1).

Original: To obtain the weights w x i ,y i , we ﬁrst compute weights w x,x i and w y,y i for each generated instance x i and y i respectively.
Revised: To obtain the weights w x i ,y i , we ﬁrst compute weights w x,x i and w y,y i for each generated instance x i and y i respectively.

Original: We use the exponential kernel to compute the weight w x,x i = exp( − D ( x, x i ) /σ 2 ) as a function of some distance D between the generated instance and the corresponding input instance.
Revised: We use the exponential kernel to compute the weight w x,x i = exp( − F ( x, x i ) /σ 2 ) as a function of some distance F between the generated instance and the corresponding input instance.

Original: D could be δ BB .
Revised: F could be δ BB .

Original: The ﬁnal weight w x i ,y i for the generated pair is then given by summing the individually computed weights of each generated data point with its respective input instance i.e. w x i ,y i =
Revised: The ﬁnal weight w x i ,y i for the generated pair is then given by summing the individually computed weights of each generated data point with its respective input instance i.e. w x i ,y i =

Original: w x,x i + w y,y i .
Revised: w x,x i + w y,y i .

Original: For Iris, the perturbation neighborhood N xy was generated for each example in the pair by sampling from a Gaussian distribution centered at that example.
Revised: For Iris, the perturbation neighborhood N xy was generated for each example in the pair by sampling from a Gaussian distribution centered at that example.

Original: The statistics for the Gaussian distribution are learned from the training set.
Revised: The statistics for the Gaussian distribution are learned from the training set.

Original: For MEPS data, perturbations for the categorical features were learned using the model discussed in the next paragraph, with a bias value of 0 . 1 .
Revised: For MEPS data, perturbations for the categorical features were learned using the model discussed in the next paragraph, with a bias value of 0 . 1 .

Original: For STS, the perturbations were generated following the LIME codebase 5 by randomly removing words from sentences.
Revised: For STS, the perturbations were generated following the LIME codebase 5 by randomly removing words from sentences.

Original: The sizes of the perturbation neighborhoods used were 100 for Iris, 200 for MEPS, and 10 for STS.
Revised: The sizes of the perturbation neighborhoods used were 100 for Iris, 200 for MEPS, and 10 for STS.

Original: The interpretable representation ( ¯ x, ¯ y ) is the same as the original features in Iris; for MEPS it involves dummy coding the categorical features, and with STS, we create a vectorized binary representation indicating just the presence or absence of words in the pair of sentences considered.
Revised: The interpretable representation ( ¯ x, ¯ y ) is the same as the original features in Iris; for MEPS it involves dummy coding the categorical features, and with STS, we create a vectorized binary representation indicating just the presence or absence of words in the pair of sentences considered.

Original: When computing perturbation neighborhood weights, D is the Manhattan distance for Iris and MEPS, whereas it is the Cosine distance for STS data.
Revised: When computing perturbation neighborhood weights, F is the Manhattan distance for Iris and MEPS, whereas it is the Cosine distance for STS data.

Original: σ 2 in exponential kernel was set to 0.5625 times the number of features for all datasets, following the default setting in LIME’s code.
Revised: σ 2 in exponential kernel was set to 0.5625 times the number of features for all datasets, following the default setting in LIME’s code.

Original: Realistic Categorical Feature Perturbation using Conditional Probability Models: For categorical features, we develop a perturbation scheme that can generate more realistic perturbations.
Revised: Realistic Categorical Feature Perturbation using Conditional Probability Models: For categorical features, we develop a perturbation scheme that can generate more realistic perturbations.

Original: For each example, we estimate the conditional probability of a feature j belonging to different categories given all the other feature values.
Revised: For each example, we estimate the conditional probability of a feature j belonging to different categories given all the other feature values.

Original: These conditional probabilities can be used to sample categories for feature j to generate perturbations.
Revised: These conditional probabilities can be used to sample categories for feature j to generate perturbations.

Original: To ensure closeness to the original category, a small constant (bias) is added to the conditional probability of the original category and re-normalized, similar to an additive smoothing scheme.
Revised: To ensure closeness to the original category, a small constant (bias) is added to the conditional probability of the original category and re-normalized, similar to an additive smoothing scheme.

Original: This can be repeated for all categorical features to obtain perturbed examples.
Revised: This can be repeated for all categorical features to obtain perturbed examples.

Original: In our experiments, the conditional probability estimator is a logistic regression model that predicts the categories of a feature j using the rest of the features in the dataset.
Revised: In our experiments, the conditional probability estimator is a logistic regression model that predicts the categories of a feature j using the rest of the features in the dataset.

Original: H D ESCRIPTIONS OF M ETRICS
Revised: H D ESCRIPTIONS OF M ETRICS

Original: Let us denote pairs of examples in a dataset D as x i
Revised: Let us denote pairs of examples in a dataset D as x i

Original: =
Revised: =

Original: ( x i 1 , x i 2 ) for i ∈ { 1 , ..., N } , the black box model prediction at x as δ BB ( x ) , the prediction of the interpretable model at x computed using the explanations obtained at z as δ z E ( x ) .
Revised: ( x i 1 , x i 2 ) for i ∈ { 1 , ..., N } , the black box model prediction at x as δ BB ( x ) , the prediction of the interpretable model at x computed using the explanations obtained at z as δ z E ( x ) .

Original: Let |D| card denote the cardinality of the set D .
Revised: Let |D| card denote the cardinality of the set D .

Original: Let K x indicate the neighbors of the pair x from D .
Revised: Let K x indicate the neighbors of the pair x from D .

Original: We compute both inﬁdelity and ﬁdelity metrics respectively based on mean absolute error (MAE) and Pearson’s r. Lower values indicate higher performance for inﬁdelity metrics, whereas higher values indicate higher performance for ﬁdelity metrics.
Revised: We compute both inﬁdelity and ﬁdelity metrics respectively based on mean absolute error (MAE) and Pearson’s r. Lower values indicate higher performance for inﬁdelity metrics, whereas higher values indicate higher performance for ﬁdelity metrics.

Original: Inﬁdelity: This is the most commonly used metric to validate the faithfulness of explanation models (Ribeiro et al., 2016).
Revised: Inﬁdelity: This is the most commonly used metric to validate the faithfulness of explanation models (Ribeiro et al., 2016).

Original: Here we deﬁne it as the MAE between the black-box and explanation model predictions across all the test points.
Revised: Here we deﬁne it as the MAE between the black-box and explanation model predictions across all the test points.

Original: For MAE, this can be denoted as
Revised: For MAE, this can be denoted as

Original: [Equation]
Revised: [Equation]

Original: Fidelity can also be computed using Pearson’s r in a similar manner.
Revised: Fidelity can also be computed using Pearson’s r in a similar manner.

Original: We differentiate the metrics discussed here with generalized (in)ﬁdelity discussed next by just calling it (in)ﬁdelity or sometimes non-generalized (
Revised: We differentiate the metrics discussed here with generalized (in)ﬁdelity discussed next by just calling it (in)ﬁdelity or sometimes non-generalized (

Original: in)ﬁdelity
Revised: in)ﬁdelity

Original: Generalized Inﬁdelity: This metric has also been used in previous works (Ramamurthy et al., 2020) to measure the generalizability of local explanations to neighboring test points.
Revised: Generalized Inﬁdelity: This metric has also been used in previous works (Ramamurthy et al., 2020) to measure the generalizability of local explanations to neighboring test points.

Original: For MAE this is deﬁned as
Revised: For MAE this is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Generalized ﬁdelity using Pearson’s r can be computed in a similar way.
Revised: Generalized ﬁdelity using Pearson’s r can be computed in a similar way.

Original: In our experiments we set K x as the 1 − nearest neighbor of the pair x , computed based on the same weighting scheme described in Appendix G.
Revised: In our experiments we set K x as the 1 − nearest neighbor of the pair x , computed based on the same weighting scheme described in Appendix G.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I M ORE Q UANTITATIVE R ESULTS
Revised: I M ORE Q UANTITATIVE R ESULTS

Original: We present comparisons of inﬁdelity in terms of MAE in Table 2.
Revised: We present comparisons of inﬁdelity in terms of MAE in Table 2.

Original: Discussions on generalized inﬁdelity are available in the main paper (Section 5.3).
Revised: Discussions on generalized inﬁdelity are available in the main paper (Section 5.3).

Original: Regarding inﬁdelity, JSLIME performs the best in all datasets, but FbFull follows closely (except for
Revised: Regarding inﬁdelity, JSLIME performs the best in all datasets, but FbFull follows closely (except for

Original: Iris data).
Revised: Iris data).

Original: This suggests that with STS and MEPS data, the black box universal sentence encoder agrees closely with the Mahalanobis distance approximation.
Revised: This suggests that with STS and MEPS data, the black box universal sentence encoder agrees closely with the Mahalanobis distance approximation.

Original: In Iris data, the black box model (Siamese neural network) probably cannot approximate the underlying similarity function well using the small number of examples provided, leading to a worse performance using our methods.
Revised: In Iris data, the black box model (Siamese neural network) probably cannot approximate the underlying similarity function well using the small number of examples provided, leading to a worse performance using our methods.

Original: However,
Revised: However,

Original: JSLIME with its unconstrained A (see Appendix E.1) is able to provide a good ﬁt.
Revised: JSLIME with its unconstrained A (see Appendix E.1) is able to provide a good ﬁt.

Original: LIME performs poorly in all cases pointing to the need to move beyond linear approximations when explaining similarity models.
Revised: LIME performs poorly in all cases pointing to the need to move beyond linear approximations when explaining similarity models.

Original: A single global approximation, GFbFull, performs reasonably well suffering only for the Iris dataset, where there is not enough examples.
Revised: A single global approximation, GFbFull, performs reasonably well suffering only for the Iris dataset, where there is not enough examples.

Original: The above story is reversed with generalized inﬁdelity metrics where our feature-based methods outperform JSLIME handsomely.
Revised: The above story is reversed with generalized inﬁdelity metrics where our feature-based methods outperform JSLIME handsomely.

Original: The performance of JSLIME is worst in Iris with generalized inﬁdelity suggesting it could be overﬁtting local explanations with this dataset.
Revised: The performance of JSLIME is worst in Iris with generalized inﬁdelity suggesting it could be overﬁtting local explanations with this dataset.

Original: We also present comparisons of ﬁdelity/generalized ﬁdelity in terms of Pearson’s r .
Revised: We also present comparisons of ﬁdelity/generalized ﬁdelity in terms of Pearson’s r .

Original: Table 3 shows the
Revised: Table 3 shows the

Original: Pearson’s r of the predictions from FbFull, FbDiag, GFbFull, LIME, and JSLIME with respect to the black box predictions.
Revised: Pearson’s r of the predictions from FbFull, FbDiag, GFbFull, LIME, and JSLIME with respect to the black box predictions.

Original: The narrative here is similar to that of inﬁdelity/generalized inﬁdelity, except for some small differences such as higher performance of FbDiag with Iris for generalized ﬁdelity.
Revised: The narrative here is similar to that of inﬁdelity/generalized inﬁdelity, except for some small differences such as higher performance of FbDiag with Iris for generalized ﬁdelity.

Original: The Pearson’s r (ﬁdelity) for the analogy explanation methods (AbE, DirSim, and PDash) and shown in Figure 5.
Revised: The Pearson’s r (ﬁdelity) for the analogy explanation methods (AbE, DirSim, and PDash) and shown in Figure 5.

Original: Just like the case of MAE (inﬁdelity) presented in Figure 4, the proposed AbE method dominates the others.
Revised: Just like the case of MAE (inﬁdelity) presented in Figure 4, the proposed AbE method dominates the others.

Original: With this metric as well, we see that for MEPS data, the performance improved slightly with the number of analogies.
Revised: With this metric as well, we see that for MEPS data, the performance improved slightly with the number of analogies.

Original: J R UNTIMES
Revised: J R UNTIMES

Original: We show the runtimes of the various explanation methods in Tables 4 and 5.
Revised: We show the runtimes of the various explanation methods in Tables 4 and 5.

Original: The compute conﬁgurations used are: A - 1 core and 64 GB RAM, B - 25 cores and 64 GB RAM, C - 50 cores and 242 GB
Revised: The compute conﬁgurations used are: A - 1 core and 64 GB RAM, B - 25 cores and 64 GB RAM, C - 50 cores and 242 GB

Original: RAM.
Revised: RAM.

Original: The black-box models are assumed to be trained and available, and the individual terms in
Revised: The black-box models are assumed to be trained and available, and the individual terms in

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (equation 2) are assumed to be pre-computed; for Table 5, runtimes reported involves only choosing the 10 analogies.
Revised: (equation 2) are assumed to be pre-computed; for Table 5, runtimes reported involves only choosing the 10 analogies.

Original: K M ORE Q UALITATIVE E XAMPLES - STS
Revised: K M ORE Q UALITATIVE E XAMPLES - STS

Original: The top three analogies for Example 3 are as follows:
Revised: The top three analogies for Example 3 are as follows:

Original: 1a)
Revised: 1a)

Original: I prefer to run the second half 1-2 minutes faster then the ﬁrst. 1b)
Revised: I prefer to run the second half 1-2 minutes faster then the ﬁrst. 1b)

Original: I would deﬁnitely go for a slightly slower ﬁrst half.
Revised: I would deﬁnitely go for a slightly slower ﬁrst half.

Original: BB distance: 0 .2a)
Revised: BB distance: 0 .2a)

Original: The pound also made progress against the dollar, reached fresh three-year highs at $1.6789.
Revised: The pound also made progress against the dollar, reached fresh three-year highs at $1.6789.

Original: 2b)
Revised: 2b)

Original: The British pound ﬂexed its muscle against the dollar, last up 1 percent at $1.6672.
Revised: The British pound ﬂexed its muscle against the dollar, last up 1 percent at $1.6672.

Original: BB distance: 0 .3a)
Revised: BB distance: 0 .3a)

Original: “I started crying and yelling at him, ‘What do you mean, what are you saying, why did you lie to me?”
Revised: “I started crying and yelling at him, ‘What do you mean, what are you saying, why did you lie to me?”

Original: ’ 3b) Gulping for air, I started crying and yelling at him, ’What do you mean?
Revised: ’ 3b) Gulping for air, I started crying and yelling at him, ’What do you mean?

Original: BB distance: 0 .
Revised: BB distance: 0 .

Original: The ﬁrst analogy is most appropriate since both sentences express the same idea (second half faster than ﬁrst half) but in different ways, similar to the input pair.
Revised: The ﬁrst analogy is most appropriate since both sentences express the same idea (second half faster than ﬁrst half) but in different ways, similar to the input pair.

Original: The second and third analogies are less appropriate because the sentences in each pair are more similar to each other than the sentences in the input pair are to each other (the BB distance for analogy 2 seems high and is likely due to not understanding the idiom “ﬂexed its muscle”).
Revised: The second and third analogies are less appropriate because the sentences in each pair are more similar to each other than the sentences in the input pair are to each other (the BB distance for analogy 2 seems high and is likely due to not understanding the idiom “ﬂexed its muscle”).

Original: Here are top analogies for the competitors for the same three examples.
Revised: Here are top analogies for the competitors for the same three examples.

Original: Example 1:
Revised: Example 1:

Original: ProtoDash - a)
Revised: ProtoDash - a)

Original: A woman is playing a
Revised: A woman is playing a

Original: , b) A man is playing a keyboard
Revised: , b) A man is playing a keyboard

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Two
Revised: Two

Original: [Figure]
Revised: [Figure]

Original: are
Revised: are

Original: [Figure]
Revised: [Figure]

Original: Example 2:
Revised: Example 2:

Original: ProtoDash - a)
Revised: ProtoDash - a)

Original: The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate group.
Revised: The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate group.

Original: b) The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate province in
Revised: b) The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate province in

Original: North America because of last week’s actions.
Revised: North America because of last week’s actions.

Original: DirSim - a) A Stage 1 episode is declared when ozone levels reach 0.20 parts per million.
Revised: DirSim - a) A Stage 1 episode is declared when ozone levels reach 0.20 parts per million.

Original: b) The federal standard for ozone is 0.12 parts per million.
Revised: b) The federal standard for ozone is 0.12 parts per million.

Original: Example 3:
Revised: Example 3:

Original: ProtoDash - a) As I wrote above, it’s hard to rate this wall.b)
Revised: ProtoDash - a) As I wrote above, it’s hard to rate this wall.b)

Original: Unlike others, I think the route is pretty well described.
Revised: Unlike others, I think the route is pretty well described.

Original: DirSim - a) Remember, from the Fleet’s point of view, the rest of the galaxy is what’s moving and experiencing time dilation, b)
Revised: DirSim - a) Remember, from the Fleet’s point of view, the rest of the galaxy is what’s moving and experiencing time dilation, b)

Original: Well, it really depends on how long he was there, and the exact speed of the Fleet.
Revised: Well, it really depends on how long he was there, and the exact speed of the Fleet.

Original: L Q UALITATIVE E XAMPLES - T ABULAR MEPS D ATA
Revised: L Q UALITATIVE E XAMPLES - T ABULAR MEPS D ATA

Original: We provide additional qualitative examples for the MEPS dataset for feature-based and Analogybased explanations.
Revised: We provide additional qualitative examples for the MEPS dataset for feature-based and Analogybased explanations.

Original: Please see MEPS feature encodings in Section L.1 for the key feature encodings used in experiments here.
Revised: Please see MEPS feature encodings in Section L.1 for the key feature encodings used in experiments here.

Original: We consider two pairs of test examples: the ﬁrst pair that has a BB distance of 0.022 (very similar) and the second pair has a BB distance of 0.584 (moderately similar).
Revised: We consider two pairs of test examples: the ﬁrst pair that has a BB distance of 0.022 (very similar) and the second pair has a BB distance of 0.584 (moderately similar).

Original: Considering the ﬁrst pair, the differences between the two examples are only along the dimensions of insurance coverage (INSCOV) and poverty category (POVCAT).
Revised: Considering the ﬁrst pair, the differences between the two examples are only along the dimensions of insurance coverage (INSCOV) and poverty category (POVCAT).

Original: The FbFull approach produces a prediction of 0 . 078 , and FbDiag’s prediction was 0 .
Revised: The FbFull approach produces a prediction of 0 . 078 , and FbDiag’s prediction was 0 .

Original: Clearly, FbFull is able to mimic the black-box model much better locally.
Revised: Clearly, FbFull is able to mimic the black-box model much better locally.

Original: The contributions to distance according to FbFull is given in Figure 6a.
Revised: The contributions to distance according to FbFull is given in Figure 6a.

Original: We see that INSCOV and
Revised: We see that INSCOV and

Original: POVCAT are picked up as signiﬁcant features for explanation.
Revised: POVCAT are picked up as signiﬁcant features for explanation.

Original: Comparisons of contributions for
Revised: Comparisons of contributions for

Original: FbFull and FbDiag are given in Figure 7.
Revised: FbFull and FbDiag are given in Figure 7.

Original: For FbFull, the contributions in Figure 7a is obtained by summing the rows or columns of the matrix in Figure 6a.
Revised: For FbFull, the contributions in Figure 7a is obtained by summing the rows or columns of the matrix in Figure 6a.

Original: FbDiag misses the mark by giving too much importance to INSCOV and hence overpredicting the black-box distance, whereas FbFull assigns reasonable importances to both INSCOV and POVCAT.
Revised: FbDiag misses the mark by giving too much importance to INSCOV and hence overpredicting the black-box distance, whereas FbFull assigns reasonable importances to both INSCOV and POVCAT.

Original: [Figure]
Revised: [Figure]

Original: We looked at 3 analogies each for AbE, DirSim, and PDash for this example, and found that the mean predicions were 0.046, 0.042, and 0.383 respectively.
Revised: We looked at 3 analogies each for AbE, DirSim, and PDash for this example, and found that the mean predicions were 0.046, 0.042, and 0.383 respectively.

Original: For this simple example, both AbE and DirSim are competitive in performance.
Revised: For this simple example, both AbE and DirSim are competitive in performance.

Original: However, we found that AbE chose pairs with more diverse set of features compared to DirSim.
Revised: However, we found that AbE chose pairs with more diverse set of features compared to DirSim.

Original: PDash chose one analogous pair with very low BB distance and two with very high BB distances which is not a desirable behavior.
Revised: PDash chose one analogous pair with very low BB distance and two with very high BB distances which is not a desirable behavior.

Original: We perform similar analyses using the second pair with a BB distance of 0.584.
Revised: We perform similar analyses using the second pair with a BB distance of 0.584.

Original: FbFull predicts a distance of 0.584, whereas FbDiag predicts a distance of 0.661.
Revised: FbFull predicts a distance of 0.584, whereas FbDiag predicts a distance of 0.661.

Original: Once again we see that FbFull more equitably perceives the contributions of the different important variables such as RACE, INSCOV,
Revised: Once again we see that FbFull more equitably perceives the contributions of the different important variables such as RACE, INSCOV,

Original: JTPAIN, and ADSMOK.
Revised: JTPAIN, and ADSMOK.

Original: FbDiag places a lot of weight on INSCOV which is probably not the overwhelmingly important contributor for the dissimilarity given differences in race and other health variables/demographics between the two data points in the pair.
Revised: FbDiag places a lot of weight on INSCOV which is probably not the overwhelmingly important contributor for the dissimilarity given differences in race and other health variables/demographics between the two data points in the pair.

Original: For this example as well, we looked at 3 analogies each for AbE, DirSim, and PDash.
Revised: For this example as well, we looked at 3 analogies each for AbE, DirSim, and PDash.

Original: The distance predictions from these methods are 0.623, 0.407, and 0.776 respectively.
Revised: The distance predictions from these methods are 0.623, 0.407, and 0.776 respectively.

Original: AbE is close to the performance of black-box whereas the other two methods over- or under-predict the distance between the data points in the pair.
Revised: AbE is close to the performance of black-box whereas the other two methods over- or under-predict the distance between the data points in the pair.

Original: In addition to the high performance AbE also produces diverse analogies whose predictions are all individually close to the black-box.
Revised: In addition to the high performance AbE also produces diverse analogies whose predictions are all individually close to the black-box.

Original: L. S ELECTED MEPS F EATURE E NCODINGS
Revised: L. S ELECTED MEPS F EATURE E NCODINGS

Original: • INSCOV=1:
Revised: • INSCOV=1:

Original: Any private insurance • INSCOV=2:
Revised: Any private insurance • INSCOV=2:

Original: Public only insurance • INSCOV=3:
Revised: Public only insurance • INSCOV=3:

Original: Uninsured • POVCAT=1: Poor/negative family income • POVCAT=3: Low family income • POVCAT=4:
Revised: Uninsured • POVCAT=1: Poor/negative family income • POVCAT=3: Low family income • POVCAT=4:

Original: Middle family income • POVCAT=5:
Revised: Middle family income • POVCAT=5:

Original: High family income • ADSMOK=-1: Inapplicable smoking status • ADSMOK=1:
Revised: High family income • ADSMOK=-1: Inapplicable smoking status • ADSMOK=1:

Original: Current smoker • ADSMOK=2:
Revised: Current smoker • ADSMOK=2:

Original: Not current smoker • RACE=1: White • RACE=0: Non-White • MNHLTH=1:
Revised: Not current smoker • RACE=1: White • RACE=0: Non-White • MNHLTH=1:

Original: Excellent (perceived mental health status)
Revised: Excellent (perceived mental health status)

Original: • MNHLTH=2: Very good (perceived mental health status)
Revised: • MNHLTH=2: Very good (perceived mental health status)

Original: [Figure]
Revised: [Figure]

Original: M U SER S TUDY S CREENSHOTS
Revised: M U SER S TUDY S CREENSHOTS

Original: We present example screenshots for the user study in Figure 9.
Revised: We present example screenshots for the user study in Figure 9.

Original: The top and middle ﬁgures are examples of analogy and feature based explanations presented in the user study.
Revised: The top and middle ﬁgures are examples of analogy and feature based explanations presented in the user study.

Original: The bottom ﬁgure is the instructions page for the user study.
Revised: The bottom ﬁgure is the instructions page for the user study.

Original: N U SER S TUDY P ARTICIPANT C OMMENTS
Revised: N U SER S TUDY P ARTICIPANT C OMMENTS

Original: • I liked the second ..... providing pairs of sentences that are analogous to the input pair, as opposed to highlighting important words, but I guess that could be subjective.
Revised: • I liked the second ..... providing pairs of sentences that are analogous to the input pair, as opposed to highlighting important words, but I guess that could be subjective.

Original: I think you hit the nail on its. head with the analogy based explanation .....
Revised: I think you hit the nail on its. head with the analogy based explanation .....

Original: just my biased view.
Revised: just my biased view.

Original: • I think the keywords make the 3 pairs easier to understand cuz it attacks it bottom up, while 3 pairs is more top down.
Revised: • I think the keywords make the 3 pairs easier to understand cuz it attacks it bottom up, while 3 pairs is more top down.

Original: So maybe best is to provide both and let user decide.
Revised: So maybe best is to provide both and let user decide.

Original: So this one was tricky: 1) You just have to base your answer on what you do know, which is what you want.
Revised: So this one was tricky: 1) You just have to base your answer on what you do know, which is what you want.

Original: You may want it, but the process given to you is what you have to work within.
Revised: You may want it, but the process given to you is what you have to work within.

Original: To me, they look dissimilar but all the explanations point to somewhat similar (so that’s what I answered).
Revised: To me, they look dissimilar but all the explanations point to somewhat similar (so that’s what I answered).

Original: (Q16)
Revised: (Q16)

Original: A similar case for Q19.
Revised: A similar case for Q19.

Original: In the case of the 3 pairs of sentences, if all of them were marked with one label only, I felt that those examples would always lead to the label used as the answer.
Revised: In the case of the 3 pairs of sentences, if all of them were marked with one label only, I felt that those examples would always lead to the label used as the answer.

Original: I ﬁnd the explanation with the difference between the sentences easier to reason about.
Revised: I ﬁnd the explanation with the difference between the sentences easier to reason about.

Original: • Identifying the speciﬁc words that marked the differences was a lot more helpful.
Revised: • Identifying the speciﬁc words that marked the differences was a lot more helpful.

Original: Parsing the examples and trying to determine how they relate to the example was pretty tricky with longer examples.
Revised: Parsing the examples and trying to determine how they relate to the example was pretty tricky with longer examples.

Original: • Some of the analogous examples were hard to map to the original samples, i.e. in what way they were analogous.
Revised: • Some of the analogous examples were hard to map to the original samples, i.e. in what way they were analogous.

Original: • The second type of explanation is clearer, but is confusing
Revised: • The second type of explanation is clearer, but is confusing

Original: for the exercise since it tells you directly the option you should choose.
Revised: for the exercise since it tells you directly the option you should choose.

Original: The ﬁrst type is better is the point is to have guesses of the answer.
Revised: The ﬁrst type is better is the point is to have guesses of the answer.

Original: For the ﬁrst type, having either a variety of outputs or multiple "somewhat similar" examples helps.
Revised: For the ﬁrst type, having either a variety of outputs or multiple "somewhat similar" examples helps.

Original: [Figure]
Revised: [Figure]

Original: • For the explanations with keywords, I just follow what the explanation said about similarity.
Revised: • For the explanations with keywords, I just follow what the explanation said about similarity.

Original: The task is for me to predict what the BB model would do, and I trust explanations are faithful to the model
Revised: The task is for me to predict what the BB model would do, and I trust explanations are faithful to the model

Original: so I just choose what the explanation says.
Revised: so I just choose what the explanation says.

Original: However, the keywords really didn’t help me understand why the model think the two are similar or dissimilar, the analogous pair did.
Revised: However, the keywords really didn’t help me understand why the model think the two are similar or dissimilar, the analogous pair did.

Original: It was hard for me to judge what the BB model would predict when analogous pairs are all dissimilar, because you can point out a lot of things that are different without telling me what you consider are similar...
Revised: It was hard for me to judge what the BB model would predict when analogous pairs are all dissimilar, because you can point out a lot of things that are different without telling me what you consider are similar...

Original: Hope this is helpful.
Revised: Hope this is helpful.

Original: • Good survey, except the explanations that were of the format: "Explanation: The sentences appear [somewhat similar]" were confusing.
Revised: • Good survey, except the explanations that were of the format: "Explanation: The sentences appear [somewhat similar]" were confusing.

Original: There is no indication in how the question was phrased to indicate the answer should be different from [somewhat similar].
Revised: There is no indication in how the question was phrased to indicate the answer should be different from [somewhat similar].

Original: • The examples seem to be more reliable than the verbal reason.
Revised: • The examples seem to be more reliable than the verbal reason.

Original: The same pair of sentences gets rated as different results due to different examples for the question.
Revised: The same pair of sentences gets rated as different results due to different examples for the question.

Original: Some key words to differentiate the sentence pair as shown in the verbal reason may not fully align with what I would suggest.
Revised: Some key words to differentiate the sentence pair as shown in the verbal reason may not fully align with what I would suggest.

Original: • Having an example of a simple, somewhat similar, and dissimilar on each problem would have been useful, as there were problems where a sentence would be provided and only one type was shown.
Revised: • Having an example of a simple, somewhat similar, and dissimilar on each problem would have been useful, as there were problems where a sentence would be provided and only one type was shown.

Original: Additionally, the problems with just the: "the AI would classify this as similar because of X, Y" were weird problems to include, as they could have acted as a calibration at the beginning of the test.
Revised: Additionally, the problems with just the: "the AI would classify this as similar because of X, Y" were weird problems to include, as they could have acted as a calibration at the beginning of the test.

Original: Instead, I found myself skipping the problem and just clicking whatever the explanation said it would classify it as to label.
Revised: Instead, I found myself skipping the problem and just clicking whatever the explanation said it would classify it as to label.

Original: • I understood the exercise better as I answered more questions so I went back to redo the initial 10.
Revised: • I understood the exercise better as I answered more questions so I went back to redo the initial 10.

Original: A slider or list of questions to quickly go back will be great.
Revised: A slider or list of questions to quickly go back will be great.

Original: Also, the initial explanation is too wordy.
Revised: Also, the initial explanation is too wordy.

Original: An demo example would have helped.
Revised: An demo example would have helped.

Original: You should group the questions based on the unique number of input pairs.
Revised: You should group the questions based on the unique number of input pairs.

Original: Dunno if all this is possible in a google form.
Revised: Dunno if all this is possible in a google form.

Original: But you have resources, I assume?
Revised: But you have resources, I assume?

Original: • I prefer to have one example each of – similar, somewhat similar, and similar.
Revised: • I prefer to have one example each of – similar, somewhat similar, and similar.

Original: Important keywords are helpful too.
Revised: Important keywords are helpful too.

Original: • It took me awhile to understand what is measure one should use for distinguishing between similar and somewhat similar.
Revised: • It took me awhile to understand what is measure one should use for distinguishing between similar and somewhat similar.

Original: Sometimes I found that even if my intuition was to pick similar, the three analogous examples said somewhat similar so in that case I went with somewhat similar as my ﬁnal answer.
Revised: Sometimes I found that even if my intuition was to pick similar, the three analogous examples said somewhat similar so in that case I went with somewhat similar as my ﬁnal answer.

Original: • I also found myself disagreeing with (a) the choice of analogous pairs and (b) the prediction for the analogous pair, so I was not sure if I should follow my intuition for what I thought was (somewhat) similar.
Revised: • I also found myself disagreeing with (a) the choice of analogous pairs and (b) the prediction for the analogous pair, so I was not sure if I should follow my intuition for what I thought was (somewhat) similar.

Original: • I found the explanation with word highlighting to be not very useful as the words highlighted didn’t seem to be indicative of the decision sometimes.
Revised: • I found the explanation with word highlighting to be not very useful as the words highlighted didn’t seem to be indicative of the decision sometimes.

Original: So I just used the machine’s prediction and my prediction to make my ﬁnal decision.
Revised: So I just used the machine’s prediction and my prediction to make my ﬁnal decision.

Original: OF ULL U SER S TUDY
Revised: O U PDATED F IGURE

Revised: Figure 2 is updated with reordered rows and columns and shown as Figure 10.

Revised: P P SEUDO C ODES

Revised: We provide pseudo codes for obtaining feature- and analogy-based similarity explanations discussed in Sections 4.1 and 4.2 respectively.

Revised: k e y b o a r d h a r p p l a y i n g m

Revised: a n

Revised: i s a keyboard harp playing man is a 0.

Revised: (a) Examplee n g a g e d o l p

Revised: h

Revised: i n t h e a s w e ll u p s c

Revised: h e m e s e t engage dolphin the as well up scheme set 0.

Revised: (b) Exampler e i w a n t g

Revised: o i n g g u e ss n e x t re i want going guess next 0.

Revised: (c) Example are ordered by decreasing order of their contributions to the distance.

Revised: Note that contribution of a word is the sum of all columns (rows) corresponding to that row (column).

Revised: Also in Figure 10c the ﬁrst word is ’re a short of the word are .

Revised: See Example sentence (b

Revised: Section

Revised: P. F EATURE -B ASED S IMILARITY E XPLANATIONS

Revised: For feature-based similarity explanations we provide the detailed pseudo code for both FbFull and

Revised: FbDiag in Algorithm 1.

Revised: Algorithm

Revised: Inputs:

Revised: Input pair to explain ( x, y ) .

Revised: Interpretable representation function ρ ( . )

Revised: that can create the interpretable representations ¯ x = ρ ( x ) , and ¯ y

Revised: = ρ ( y ) .

Revised: Black box model δ BB ( x, y ) .

Revised: Method to create the perturbation neighborhood N xy for the input pair.

Revised: Exponential kernel parameter σ for computing weights between original and perturbed pair.

Revised: Distance function F to be used in the exponential kernel.

Revised: [Table]

Revised: Step 1: Create the perturbation neighborhood N xy for the input pair ( x, y ) containing the pairs ( x i , y i ) .

Revised: Step 2: Create black box prediction δ BB ( x i , y i ) for pairs in N xy .

Revised: Step 3:

Revised: Create corresponding interpretable representations (¯ x i , ¯ y i ) using the interpretable representation function ρ ( . ) .

Revised: Step 4: Compute weights for each component in the pair as w x,x i = exp( − F ( x, x i ) /σ 2 ) , w y,y i = exp( − F

Revised: ( y, y i )

Revised: /σ 2 ) .

Revised: Total weight for the pair w x

Revised: i ,y i =

Revised: w x,x i + w y,y i .

Revised: Step 5:

Revised: Using (1) obtain A that can be used to explain the black box function δ BB ( ., . )

Revised: at ( x, y ) .

Revised: This corresponds to the FbFull method.

Revised: Note:

Revised: If A is constrained to be diagonal, (1) reduces to a least squares with non-negativity constraint as discussed in Section 4.1.

Revised: This corresponds to the FbDiag method.

Revised: Output: Matrix A (full or diagonal) that can be used to explain the black box function δ BB ( ., . )

Revised: at ( x, y ) .

Revised: P. A NALOGY -B ASED S IMILARITY E XPLANATIONS

Revised: For analogy-based similarity explanations we provide the detailed pseudo code for the greedy version discussed in Section C in Algorithm 1.

Revised: Algorithm 2: Analogy-Based Similarity Explanations.

Revised: Inputs:

Revised: Input pair to explain x = ( x 1 , x 2 ) .

Revised: Pairs of examples in the dataset z

Revised: i

Revised: =

Revised: ( z i 1 , z

Revised: i 2 ) for i ∈ { 1 , ..., N } to draw analogies from.

Revised: These pairs constitute the candidate set C . 3.

Revised: Black box model δ BB ( x, y ) .

Revised: Embedding function φ ( . ) used to compute ﬁrst part, D

Revised: ( z i , x ) , of the analogy closeness term, G ( z i , x ) (see equation 3–equation 4).

Revised: Distance predicted by the feature-based explainer

Revised: δ I ( x ) used in equation 3.

Revised: Use Algorithmto obtain feature-based explanations for x . 6.

Revised: Hyperparameters λ 1 (contribution of analogy closeness term), λ 2 (contribution of diversity term), and α (contribution of feature-based explainer) used in equation 2–equation 4).

Revised: Number of analogies to be chosen, k . 8.

Revised: Analogous pairs set S = ∅ .

Revised: Method: do

Revised: [Equation]

Revised: S ← S ∪ { a } while | S | < k ;

Revised: Output: Set of pairs analogous to x given by the set S .

Revised: Q F ULL U SER S TUDY

Original: The printout of the full user study is attached from the next page.
Revised: The printout of the full user study is attached from the next page.

Original: [Table]
Revised: [Table]

Original: 1) A group of men play soccer on the beach.
Revised: 1) A group of men play soccer on the beach.

Original: A group of boys are playing soccer on the beach.
Revised: A group of boys are playing soccer on the beach.

Original: [Table]
Revised: [Table]

Original: AI Explanations for Pairwise Similarity Models
Revised: AI Explanations for Pairwise Similarity Models

Original: PLEASE READ CAREFULLY
Revised: PLEASE READ CAREFULLY

Original: Thank you so much for volunteering to take this survey.
Revised: Thank you so much for volunteering to take this survey.

Original: No information other than that you choose to provide will be recorded in this survey.
Revised: No information other than that you choose to provide will be recorded in this survey.

Original: There are 30 questions and it should take around 15 minutes to complete the survey.
Revised: There are 30 questions and it should take around 15 minutes to complete the survey.

Original: At the end you can also leave additional feedback.
Revised: At the end you can also leave additional feedback.

Original: Below are the details of the survey.
Revised: Below are the details of the survey.

Original: A black-box (BB) AI model was trained to predict similarity between pairs of sentences.
Revised: A black-box (BB) AI model was trained to predict similarity between pairs of sentences.

Original: For example, given a pair sentences "A boy ate breakfast" and "A girl ate breakfast", the BB model will predict a similarity value between 0 and 1.
Revised: For example, given a pair sentences "A boy ate breakfast" and "A girl ate breakfast", the BB model will predict a similarity value between 0 and 1.

Original: For the purposes of this survey, we bucket the predictions of the model into three categories - similar, somewhat similar and dissimilar.
Revised: For the purposes of this survey, we bucket the predictions of the model into three categories - similar, somewhat similar and dissimilar.

Original: Given a pair of sentences, your task is to guess which of the three categories the BB model would predict based on the provided explanations.
Revised: Given a pair of sentences, your task is to guess which of the three categories the BB model would predict based on the provided explanations.

Original: You will be shown the output of different (anonymized) AI explanation methods.
Revised: You will be shown the output of different (anonymized) AI explanation methods.

Original: Each explanation method provides one of two types of explanations.
Revised: Each explanation method provides one of two types of explanations.

Original: The first explanation type highlights the important words in the sentences that it thinks led to the BB model's prediction along with its assessment of what the similarity level is likely to be.
Revised: The first explanation type highlights the important words in the sentences that it thinks led to the BB model's prediction along with its assessment of what the similarity level is likely to be.

Original: The other type of explanation provides three pairs of sentences that are analogous to the given input pair.
Revised: The other type of explanation provides three pairs of sentences that are analogous to the given input pair.

Original: In all cases the question is: HOW SIMILAR ARE THE INPUT SENTENCES ACCORDING TO THE BB AI MODEL BASED ON
Revised: In all cases the question is: HOW SIMILAR ARE THE INPUT SENTENCES ACCORDING TO THE BB AI MODEL BASED ON

Original: THE
Revised: THE

Original: PROVIDED EXPLANATION(S)?
Revised: PROVIDED EXPLANATION(S)?

Original: [Table]
Revised: [Table]

Original: Explanations
Revised: Explanations

Original: [Table]
Revised: [Table]

Original: https://docs.google.com/forms/u/0/d/1lGmXZZcf4LFlMeUzmN5GALCC1R0FUiOyMVM3Q...
Revised: https://docs.google.com/forms/u/0/d/1lGmXZZcf4LFlMeUzmN5GALCC1R0FUiOyMVM3Q...

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: of
Revised: of

Original: a) Two people in snowsuits laying in the snow making snow angels.
Revised: a) Two people in snowsuits laying in the snow making snow angels.

Original: b) Two children lying in the snow making snow angels.
Revised: b) Two children lying in the snow making snow angels.

Original: 2nd analogous pair (similar): a) A sad man is jumping over a small stream to meet his companion on the other side.
Revised: 2nd analogous pair (similar): a) A sad man is jumping over a small stream to meet his companion on the other side.

Original: b) A man is jumping over a stream to meet his companion on the other side.
Revised: b) A man is jumping over a stream to meet his companion on the other side.

Original: 3rd analogous pair (similar): a) A woman puts flour on a piece of meat.
Revised: 3rd analogous pair (similar): a) A woman puts flour on a piece of meat.

Original: b) A woman is putting flour onto some meat.
Revised: b) A woman is putting flour onto some meat.

Original: Question
Revised: Question

Original: 2 of
Revised: 2 of

Original: [Table]
Revised: [Table]

Original: 1) A group of men play soccer on the beach.
Revised: 1) A group of men play soccer on the beach.

Original: A group of boys are playing soccer on the beach.
Revised: A group of boys are playing soccer on the beach.

Original: [Table]
Revised: [Table]

Original: 2 of10/2/21, 4:11 PM
Revised: 2 of10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: [Table]
Revised: [Table]

Original: 2nd analogous pair (dissimilar) a)
Revised: 2nd analogous pair (dissimilar) a)

Original: The man is aiming a gun.
Revised: The man is aiming a gun.

Original: b) A boy is playing on a toy phone.
Revised: b) A boy is playing on a toy phone.

Original: 3rd analogous pair (somewhat similar) a)
Revised: 3rd analogous pair (somewhat similar) a)

Original: The religious people are enjoying the outdoors.
Revised: The religious people are enjoying the outdoors.

Original: b)
Revised: b)

Original: The group of people are enjoying the outdoors.
Revised: The group of people are enjoying the outdoors.

Original: Question 3 of
Revised: Question 3 of

Original: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below
Revised: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below

Original: [Table]
Revised: [Table]

Original: 1) A group of men play soccer on the beach.
Revised: 1) A group of men play soccer on the beach.

Original: A group of boys are playing soccer on the beach.
Revised: A group of boys are playing soccer on the beach.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a) A group of men playing soccer in a stadium full of people.
Revised: 1st analogous pair (similar) a) A group of men playing soccer in a stadium full of people.

Original: b) A group of men playing soccer in a stadium full of people.
Revised: b) A group of men playing soccer in a stadium full of people.

Original: 2nd analogous pair (somewhat similar) a) A pair of men walk along the beach.
Revised: 2nd analogous pair (somewhat similar) a) A pair of men walk along the beach.

Original: b) Two boys in black swimming trunks are holding another boy by his arms and legs on a beach.
Revised: b) Two boys in black swimming trunks are holding another boy by his arms and legs on a beach.

Original: [Table]
Revised: [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval
Revised: Mark only one oval

Original: [Table]
Revised: [Table]

Original: Question 4 of
Revised: Question 4 of

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a harp.
Revised: 1) A man is playing a harp.

Original: A man is playing a keyboard.
Revised: A man is playing a keyboard.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (somewhat similar) a) A woman is playing the flute.
Revised: 1st analogous pair (somewhat similar) a) A woman is playing the flute.

Original: b) A man is playing a keyboard.
Revised: b) A man is playing a keyboard.

Original: 2nd analogous pair (somewhat similar) a)
Revised: 2nd analogous pair (somewhat similar) a)

Original: A guy is playing hackysack b)
Revised: A guy is playing hackysack b)

Original: A man is playing a key-board.
Revised: A man is playing a key-board.

Original: 3rd analogous pair (similar) a) Women are running.
Revised: 3rd analogous pair (similar) a) Women are running.

Original: b) Two women are running.
Revised: b) Two women are running.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say Question
Revised: Similar Somewhat similar Dissimilar Can't say Question

Original: 5 of
Revised: 5 of

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a harp.
Revised: 1) A man is playing a harp.

Original: A man is playing a keyboard.
Revised: A man is playing a keyboard.

Original: [Table]
Revised: [Table]

Original: 1st
Revised: 1st

Original: [Table]
Revised: [Table]

Original: pair (similar) a) Women are running.
Revised: pair (similar) a) Women are running.

Original: b) Two women are running.
Revised: b) Two women are running.

Original: 2nd analogous pair (somewhat similar) a) A man is playing a flute.
Revised: 2nd analogous pair (somewhat similar) a) A man is playing a flute.

Original: b) A man plays a keyboard.
Revised: b) A man plays a keyboard.

Original: 2nd analogous pair (dissimilar) a)
Revised: 2nd analogous pair (dissimilar) a)

Original: Police believe Wilson then shot
Revised: Police believe Wilson then shot

Original: [Table]
Revised: [Table]

Original: Mae Robinson once in the head before turning the gun on herself.
Revised: Mae Robinson once in the head before turning the gun on herself.

Original: b) "What they have done is a thinly veiled attempt to do an end run around the Constitution," she said.
Revised: b) "What they have done is a thinly veiled attempt to do an end run around the Constitution," she said.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say 6.
Revised: Similar Somewhat similar Dissimilar Can't say 6.

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Questionof
Revised: Questionof

Original: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below
Revised: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a harp.
Revised: 1) A man is playing a harp.

Original: A man is playing a keyboard.
Revised: A man is playing a keyboard.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: Somewhat similar
Revised: Somewhat similar

Original: Dissimilar
Revised: Dissimilar

Original: Can't say
Revised: Can't say

Original: Question 7 of
Revised: Question 7 of

Original: [Table]
Revised: [Table]

Original: A woman is swimming underwater.
Revised: A woman is swimming underwater.

Original: A man is slicing some carrots.
Revised: A man is slicing some carrots.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (dissimilar) a) A person wearing scuba gear is swimming underwater water.
Revised: 1st analogous pair (dissimilar) a) A person wearing scuba gear is swimming underwater water.

Original: b) A man is cutting up carrots.
Revised: b) A man is cutting up carrots.

Original: 2nd analogous pair (somewhat similar) a)
Revised: 2nd analogous pair (somewhat similar) a)

Original: A woman is relaxing in a bath tub.
Revised: A woman is relaxing in a bath tub.

Original: b) A woman is slicing carrot.
Revised: b) A woman is slicing carrot.

Original: 3rd analogous pair (dissimilar) a) a young girl smiling with her head upside down b) I have a few Nikon lenses, and a Sigma 10-20mm for super-wide shots.
Revised: 3rd analogous pair (dissimilar) a) a young girl smiling with her head upside down b) I have a few Nikon lenses, and a Sigma 10-20mm for super-wide shots.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Questionof
Revised: Questionof

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Questionof
Revised: Questionof

Original: [Table]
Revised: [Table]

Original: A woman is swimming underwater.
Revised: A woman is swimming underwater.

Original: A man
Revised: A man

Original: [Table]
Revised: [Table]

Original: 8 of10/2/21, 4:11 PM
Revised: 8 of10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say 10.
Revised: Similar Somewhat similar Dissimilar Can't say 10.

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question 10 of 
Revised: Question 10 of

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a guitar.
Revised: 1) A man is playing a guitar.

Original: A man is playing a trumpet.
Revised: A man is playing a trumpet.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: Somewhat similar
Revised: Somewhat similar

Original: Dissimilar
Revised: Dissimilar

Original: Can't say
Revised: Can't say

Original: Questionof
Revised: Questionof

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a guitar.
Revised: 1) A man is playing a guitar.

Original: A man is playing a trumpet.
Revised: A man is playing a trumpet.

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: 12 of
Revised: 12 of

Original: [Table]
Revised: [Table]

Original: 1) A man is playing a guitar.
Revised: 1) A man is playing a guitar.

Original: A man is playing a trumpet.
Revised: A man is playing a trumpet.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: guitar
Revised: guitar

Original: [Table] [Table]
Revised: [Table] [Table]

Original: is playing a
Revised: is playing a

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 10 of10/2/21, 4:11 PM
Revised: 10 of10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: of
Revised: of

Original: [Table]
Revised: [Table]

Original: 1) A man is eating a food.
Revised: 1) A man is eating a food.

Original: A man is eating a piece of bread.
Revised: A man is eating a piece of bread.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question
Revised: Question

Original: 14 of
Revised: 14 of

Original: [Table]
Revised: [Table]

Original: 1) A man is eating a food.
Revised: 1) A man is eating a food.

Original: A man is eating a piece of bread.
Revised: A man is eating a piece of bread.

Original: [Table]
Revised: [Table]

Original: ===
Revised: ===

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a) An animal is chewing on something.
Revised: 1st analogous pair (similar) a) An animal is chewing on something.

Original: b) An animal is chewing on a key chain.
Revised: b) An animal is chewing on a key chain.

Original: 2nd analogous pair (somewhat similar) a) The Hare Psychopathy Checklist is often used to assess psychopathy in clinical settings.
Revised: 2nd analogous pair (somewhat similar) a) The Hare Psychopathy Checklist is often used to assess psychopathy in clinical settings.

Original: b) From an article entitled Can You Call a 9-Year-Old a Psychopath?
Revised: b) From an article entitled Can You Call a 9-Year-Old a Psychopath?

Original: 3rd analogous pair (similar) a) A player bounces a ball.
Revised: 3rd analogous pair (similar) a) A player bounces a ball.

Original: b) A player catching a ball.
Revised: b) A player catching a ball.

Original: [Table]
Revised: [Table]

Original: PM
Revised: PM

Original: Mark only one oval
Revised: Mark only one oval

Original: [Table]
Revised: [Table]

Original: Can
Revised: Can

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: 15 of
Revised: 15 of

Original: [Table]
Revised: [Table]

Original: 1) A man is eating a food.
Revised: 1) A man is eating a food.

Original: A man is eating a piece of bread.
Revised: A man is eating a piece of bread.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a) An animal is chewing on something.
Revised: 1st analogous pair (similar) a) An animal is chewing on something.

Original: b) An animal is chewing on a key chain.
Revised: b) An animal is chewing on a key chain.

Original: 2nd analogous pair (dissimilar) a) A girl is dancing in a cage.
Revised: 2nd analogous pair (dissimilar) a) A girl is dancing in a cage.

Original: b) A onion is being chopped.
Revised: b) A onion is being chopped.

Original: 3rd analogous pair (somewhat similar) a) A man shoots a basket.
Revised: 3rd analogous pair (somewhat similar) a) A man shoots a basket.

Original: b) A man is playing a guitar.
Revised: b) A man is playing a guitar.

Original: [Table]
Revised: [Table]

Original: Mark only one oval
Revised: Mark only one oval

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question
Revised: Question

Original: 16 of
Revised: 16 of

Original: [Table]
Revised: [Table]

Original: 1) You just have to base your answer on what you do know, which is what you want.
Revised: 1) You just have to base your answer on what you do know, which is what you want.

Original: You may want it, but the process given to you is what you have to work within.
Revised: You may want it, but the process given to you is what you have to work within.

Original: ==
Revised: ==

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Original: [Equation] = [Equation]
Revised: = =

Original: == = =
Revised: =

Original: =

Original: =

Original: =

Original: =

Original: =

Original: =

Original: =

Original: =

Original: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: = = = ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (somewhat similar) a)
Revised: 1st analogous pair (somewhat similar) a)

Original: In English, certainly the most common use of do is Do-Support.
Revised: In English, certainly the most common use of do is Do-Support.

Original: b) In traditional grammar, the word doing is a participle in all your examples.
Revised: b) In traditional grammar, the word doing is a participle in all your examples.

Original: 2nd analogous pair (somewhat similar) a) Stocking is very subjective to the specific inhabitants as well as the setup and your aquarium experience.
Revised: 2nd analogous pair (somewhat similar) a) Stocking is very subjective to the specific inhabitants as well as the setup and your aquarium experience.

Original: b) You'll always want to add fish into a tank slowly.
Revised: b) You'll always want to add fish into a tank slowly.

Original: 3rd analogous pair (somewhat similar) a) A man is on a rooftop b)
Revised: 3rd analogous pair (somewhat similar) a) A man is on a rooftop b)

Original: A man is holding a microphone in a room.
Revised: A man is holding a microphone in a room.

Original: [Table]
Revised: [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: [Table]
Revised: [Table]

Original: Question of
Revised: Question of

Original: [Table]
Revised: [Table]

Original: 1) You just have to base your answer on what you do know, which is what you want.
Revised: 1) You just have to base your answer on what you do know, which is what you want.

Original: You may want it, but the process given to you is what you have to work within.
Revised: You may want it, but the process given to you is what you have to work within.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (dissimilar) a) Even though the question has been answered I would like to add my answer .
Revised: 1st analogous pair (dissimilar) a) Even though the question has been answered I would like to add my answer .

Original: b) I don't know if I should say this, but if your job is making you anxious.
Revised: b) I don't know if I should say this, but if your job is making you anxious.

Original: 2nd analogous pair (dissimilar) a) I will advise exactly the contrary of what bravo just said in another answer : go for A !
Revised: 2nd analogous pair (dissimilar) a) I will advise exactly the contrary of what bravo just said in another answer : go for A !

Original: b) You could defer admission, but it's a little unusual to defer for a year.
Revised: b) You could defer admission, but it's a little unusual to defer for a year.

Original: 3rd analogous pair (dissimilar) a) People who are good at the philosophy of mathematics are mathematicians and not philosophers.
Revised: 3rd analogous pair (dissimilar) a) People who are good at the philosophy of mathematics are mathematicians and not philosophers.

Original: b)
Revised: b)

Original: My motivation is that I like to look at things from their roots and go up.
Revised: My motivation is that I like to look at things from their roots and go up.

Original: [Table]
Revised: [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: [Table]
Revised: [Table]

Original: Dissimilar Can
Revised: Dissimilar Can

Original: [Table]
Revised: [Table]

Original: t say
Revised: t say

Original: Question of
Revised: Question of

Original: [Table]
Revised: [Table]

Original: 1) You just have to base your answer on what you do know, which is what you want.
Revised: 1) You just have to base your answer on what you do know, which is what you want.

Original: You may want it, but the process given to you is what you have to work within. =
Revised: You may want it, but the process given to you is what you have to work within. =

Original: =
Revised: =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (dissimilar) a) Since you're being kind of vague here, I can only give you a vague answer.
Revised: 1st analogous pair (dissimilar) a) Since you're being kind of vague here, I can only give you a vague answer.

Original: b) Look at this: http://www.capewrathtrail.co.uk/
Revised: b) Look at this: http://www.capewrathtrail.co.uk/

Original: You could do it in stages.
Revised: You could do it in stages.

Original: 2nd analogous pair (dissimilar) a) You can always ask, then it is the choice of the author to accept or not.
Revised: 2nd analogous pair (dissimilar) a) You can always ask, then it is the choice of the author to accept or not.

Original: b)
Revised: b)

Original: The short answer is that yes, you can do this.
Revised: The short answer is that yes, you can do this.

Original: 3rd analogous pair (dissimilar) a) You have to define the problem before attempting a solution.
Revised: 3rd analogous pair (dissimilar) a) You have to define the problem before attempting a solution.

Original: b) As I understand it, nothing that can be changed, or broken down into smaller parts is inherently real.
Revised: b) As I understand it, nothing that can be changed, or broken down into smaller parts is inherently real.

Original: [Table]
Revised: [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: [Table]
Revised: [Table]

Original: Dissimilar Can
Revised: Dissimilar Can

Original: [Table]
Revised: [Table]

Original: say
Revised: say

Original: Question 19 of
Revised: Question 19 of

Original: [Table]
Revised: [Table]

Original: 1) You should just ask your boss what he wants you to do.
Revised: 1) You should just ask your boss what he wants you to do.

Original: You should listen to your boss, because you're not paid to tell the boss what to do.
Revised: You should listen to your boss, because you're not paid to tell the boss what to do.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a) two young girls hug each other in the grass.
Revised: 1st analogous pair (similar) a) two young girls hug each other in the grass.

Original: b) two young girls holding each other on the grassy ground 2nd analogous pair (somewhat similar) a) Artists are worried the plan would harm those who need help most - performers who have a difficult time lining up shows.
Revised: b) two young girls holding each other on the grassy ground 2nd analogous pair (somewhat similar) a) Artists are worried the plan would harm those who need help most - performers who have a difficult time lining up shows.

Original: b) The artists say the plan will harm French culture and punish those who need help most - performers who have a hard time lining up work.
Revised: b) The artists say the plan will harm French culture and punish those who need help most - performers who have a hard time lining up work.

Original: 3rd analogous pair (somewhat similar) a)
Revised: 3rd analogous pair (somewhat similar) a)

Original: The villains I absolutely hate are selfish and self-serving, but they are also cowards.
Revised: The villains I absolutely hate are selfish and self-serving, but they are also cowards.

Original: b) A villain you want to take down is, at his/her core, someone who does not care about the suffering of others.
Revised: b) A villain you want to take down is, at his/her core, someone who does not care about the suffering of others.

Original: [Table]
Revised: [Table]

Original: 17 of10/2/21, 4:11 PM
Revised: 17 of10/2/21, 4:11 PM

Original: Mark only
Revised: Mark only

Original: one oval
Revised: one oval

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question
Revised: Question

Original: 20 of
Revised: 20 of

Original: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below
Revised: Please choose below what the BB model will predict for the following input sentence pair based on the explanation provided below

Original: [Table]
Revised: [Table]

Original: 1) You should just ask your boss what he wants you to do.
Revised: 1) You should just ask your boss what he wants you to do.

Original: You should listen to your boss, because you're not paid to tell the boss what to do.
Revised: You should listen to your boss, because you're not paid to tell the boss what to do.

Original: =
Revised: =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a) I don't know if I should say this, but if your job is making you anxious.
Revised: 1st analogous pair (similar) a) I don't know if I should say this, but if your job is making you anxious.

Original: b) I don't know if I should say this, but if your job is making you anxious.
Revised: b) I don't know if I should say this, but if your job is making you anxious.

Original: 2nd analogous pair (dissimilar) a) As Eugene suggests, try to figure out what you want to achieve.
Revised: 2nd analogous pair (dissimilar) a) As Eugene suggests, try to figure out what you want to achieve.

Original: b) A woman supervisor is instructing the male workers.
Revised: b) A woman supervisor is instructing the male workers.

Original: 3rd analogous pair (dissimilar) a) A woman supervisor is instructing the male workers.
Revised: 3rd analogous pair (dissimilar) a) A woman supervisor is instructing the male workers.

Original: b) Basically, the rule is that you must play the ball, and not the man, until someone catches the ball.
Revised: b) Basically, the rule is that you must play the ball, and not the man, until someone catches the ball.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say 21.
Revised: Similar Somewhat similar Dissimilar Can't say 21.

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question 21 of 
Revised: Question 21 of

Original: [Table]
Revised: [Table]

Original: 1) You should just ask your boss what he wants you to do.
Revised: 1) You should just ask your boss what he wants you to do.

Original: You should listen to your boss, because you're not paid to tell the boss what to do. =
Revised: You should listen to your boss, because you're not paid to tell the boss what to do. =

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: ===
Revised: ===

Original: [Table]
Revised: [Table]

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: ===
Revised: ===

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: Somewhat similar
Revised: Somewhat similar

Original: Dissimilar
Revised: Dissimilar

Original: Can't say
Revised: Can't say

Original: Question
Revised: Question

Original: 22 of
Revised: 22 of

Original: [Table]
Revised: [Table]

Original: 1) You need to read a lot to know what you like and what you don't.
Revised: 1) You need to read a lot to know what you like and what you don't.

Original: Yes, you should create a portfolio site to showcase what you can do and what you've done.
Revised: Yes, you should create a portfolio site to showcase what you can do and what you've done.

Original: =
Revised: =

Original: =
Revised: =

Original: == ==
Revised: == ==

Original: == ==
Revised: == ==

Original: ==
Revised: ==

Original: == ===
Revised: == ===

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (dissimilar) a) I'd say you need to distract yourself from the bigger picture.
Revised: 1st analogous pair (dissimilar) a) I'd say you need to distract yourself from the bigger picture.

Original: b) No, you don't need to have taken classes or earned a degree in your area.
Revised: b) No, you don't need to have taken classes or earned a degree in your area.

Original: 2nd analogous pair (dissimilar) a)
Revised: 2nd analogous pair (dissimilar) a)

Original: A little girl and boy are reading books.
Revised: A little girl and boy are reading books.

Original: b) For completeness, Apple's Pages has quite a few nice poster layouts.
Revised: b) For completeness, Apple's Pages has quite a few nice poster layouts.

Original: 3rd analogous pair (dissimilar) a)
Revised: 3rd analogous pair (dissimilar) a)

Original: The info you've been given in the previous answers is good.
Revised: The info you've been given in the previous answers is good.

Original: b) You'll need to check the particular policies of each publisher to see what is allowed and what is not allowed.
Revised: b) You'll need to check the particular policies of each publisher to see what is allowed and what is not allowed.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval
Revised: Mark only one oval

Original: [Table]
Revised: [Table]

Original: Question 23 of
Revised: Question 23 of

Original: 1) You need to read a lot to know what you like and what you don't.
Revised: 1) You need to read a lot to know what you like and what you don't.

Original: Yes, you should create a portfolio site to showcase what you can do and what you've done.
Revised: Yes, you should create a portfolio site to showcase what you can do and what you've done.

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: of
Revised: of

Original: [Table]
Revised: [Table]

Original: 1) You need to read a lot to know what you like and what you don't.
Revised: 1) You need to read a lot to know what you like and what you don't.

Original: Yes, you should create a portfolio site to showcase what you can do and what you've done.
Revised: Yes, you should create a portfolio site to showcase what you can do and what you've done.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 10/2/21, 4:11 PM
Revised: 10/2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: [Table]
Revised: [Table]

Original: Question
Revised: Question

Original: 25 of
Revised: 25 of

Original: [Table]
Revised: [Table]

Original: BB model
Revised: BB model

Original: [Table]
Revised: [Table]

Original: predict for
Revised: predict for

Original: [Table]
Revised: [Table]

Original: following input sentence pair
Revised: following input sentence pair

Original: [Table]
Revised: [Table]

Original: explanation
Revised: explanation

Original: [Table]
Revised: [Table]

Original: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.
Revised: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.

Original: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls. 
Revised: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls. = = =

Revised: =

Revised: =

Original: = 
Revised: = = = =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Revised: =

Original: = = 
Revised: = = =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: == = =
Revised: =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say 26.
Revised: Similar Somewhat similar Dissimilar Can't say 26.

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Question
Revised: Question

Original: 26 of
Revised: 26 of

Original: [Table]
Revised: [Table]

Original: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.
Revised: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.

Original: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls.
Revised: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls.

Original: =
Revised: =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: Question 27 of
Revised: Question 27 of

Original: [Table]
Revised: [Table]

Original: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.
Revised: 1) As well as the dolphin scheme, the chaos has allowed foreign companies to engage in damaging logging and fishing operations without proper monitoring or export controls.

Original: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls.
Revised: 2) Internal chaos has allowed foreign companies to set up damaging commercial logging and fishing operations without proper monitoring or export controls.

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (similar) a)
Revised: 1st analogous pair (similar) a)

Original: The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate group.
Revised: The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate group.

Original: b) The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate province in North America because of last week's actions.
Revised: b) The American Anglican Council, which represents Episcopalian conservatives, said it will seek authorization to create a separate province in North America because of last week's actions.

Original: 2nd analogous pair (somewhat similar) a) A Stage 1 episode is declared when ozone levels reach 0.20 parts per million.
Revised: 2nd analogous pair (somewhat similar) a) A Stage 1 episode is declared when ozone levels reach 0.20 parts per million.

Original: b) The federal standard for ozone is 0.12 parts per million.
Revised: b) The federal standard for ozone is 0.12 parts per million.

Original: 3rd analogous pair (similar) a) Singapore is already the United States' 12th-largest trading partner, with two-way trade totaling more than $34 billion.
Revised: 3rd analogous pair (similar) a) Singapore is already the United States' 12th-largest trading partner, with two-way trade totaling more than $34 billion.

Original: b) Although a small city-state, Singapore is the 12th-largest trading partner of the United States, with trade volume of $33.4 billion last year.
Revised: b) Although a small city-state, Singapore is the 12th-largest trading partner of the United States, with trade volume of $33.4 billion last year.

Original: [Table]
Revised: [Table]

Original: 2/21, 4:11 PM
Revised: 2/21, 4:11 PM

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say 28.
Revised: Similar Somewhat similar Dissimilar Can't say 28.

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar Somewhat similar Dissimilar Can't say
Revised: Similar Somewhat similar Dissimilar Can't say

Original: Questionof
Revised: Questionof

Original: [Table]
Revised: [Table]

Original: 1) It depends on what you want to do next, and where you want to do it.
Revised: 1) It depends on what you want to do next, and where you want to do it.

Original: I guess it depends on what you're going to do.
Revised: I guess it depends on what you're going to do.

Original: [Table]
Revised: [Table]

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: Somewhat similar
Revised: Somewhat similar

Original: Dissimilar
Revised: Dissimilar

Original: Can't say
Revised: Can't say

Original: Question of
Revised: Question of

Original: [Table]
Revised: [Table]

Original: 1) It depends on what you want to do next, and where you want to do it.
Revised: 1) It depends on what you want to do next, and where you want to do it.

Original: I guess it depends on what you're going to do.
Revised: I guess it depends on what you're going to do.

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: 1st analogous pair (somewhat similar) a) I prefer to run the second half 1-2 minutes faster then the first.
Revised: 1st analogous pair (somewhat similar) a) I prefer to run the second half 1-2 minutes faster then the first.

Original: b) I would definitely go for a slightly slower first half.
Revised: b) I would definitely go for a slightly slower first half.

Original: 2nd analogous pair (somewhat similar) a)
Revised: 2nd analogous pair (somewhat similar) a)

Original: The pound also made progress against the dollar, reached fresh three-year highs at $1.6789.
Revised: The pound also made progress against the dollar, reached fresh three-year highs at $1.6789.

Original: b) The British pound flexed its muscle against the dollar, last up 1 percent at $1.6672.
Revised: b) The British pound flexed its muscle against the dollar, last up 1 percent at $1.6672.

Original: 3rd analogous pair (somewhat similar) a) "I started crying and yelling at him, 'What do you mean, what are you saying, why did you lie to me?'"
Revised: 3rd analogous pair (somewhat similar) a) "I started crying and yelling at him, 'What do you mean, what are you saying, why did you lie to me?'"

Original: b) Gulping for air, I started crying and yelling at him, 'What do you mean?
Revised: b) Gulping for air, I started crying and yelling at him, 'What do you mean?

Original: [Table]
Revised: [Table]

Original: 1) It depends on what you want to do next, and where you want to do it.
Revised: 1) It depends on what you want to do next, and where you want to do it.

Original: [Table]
Revised: [Table]

Original: Mark only one oval.
Revised: Mark only one oval.

Original: Similar
Revised: Similar

Original: Somewhat similar
Revised: Somewhat similar

Original: Dissimilar
Revised: Dissimilar

Original: Can't say
Revised: Can't say

Original: Question
Revised: Question

Original: 30 of
Revised: 30 of

Original: 2) I guess it depends on what you're going to do.
Revised: 2) I guess it depends on what you're going to do.

Original: [Table]
Revised: [Table]

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: = =
Revised: = =

Original: =
Revised: =

Original: ==
Revised: ==

Original: [Table]
Revised: [Table]

Original: 1st
Revised: 1st

Original: [Table]
Revised: [Table]

Original: ) a) As I wrote above, it's hard to rate this wall.
Revised: ) a) As I wrote above, it's hard to rate this wall.

Original: b) Unlike others, I think the route is pretty well described.
Revised: b) Unlike others, I think the route is pretty well described.

Original: 2nd analogous pair (somewhat similar) a) Remember, from the Fleet's point of view, the rest of the galaxy is what's moving and experiencing time dilation.
Revised: 2nd analogous pair (somewhat similar) a) Remember, from the Fleet's point of view, the rest of the galaxy is what's moving and experiencing time dilation.

Original: b)
Revised: b)

Original: Well, it really depends on how long he was there, and the exact speed of the Fleet.
Revised: Well, it really depends on how long he was there, and the exact speed of the Fleet.

Original: 3rd analogous pair (similar) a) Deaths in rollover crashes accounted for 82 percent of the number of traffic deaths in 2002, the agency says.
Revised: 3rd analogous pair (similar) a) Deaths in rollover crashes accounted for 82 percent of the number of traffic deaths in 2002, the agency says.

Original: b) Fatalities in rollover crashes accounted for 82 percent of the increase in 2002, NHTSA said.
Revised: b) Fatalities in rollover crashes accounted for 82 percent of the increase in 2002, NHTSA said.

Original: [Table]
Revised: [Table]

Original: Please provide any (optional) feedback you might have.
Revised: Please provide any (optional) feedback you might have.

Original: For example, did you find any of the explanations helpful, and if so, why?
Revised: For example, did you find any of the explanations helpful, and if so, why?

Original: Did you prefer one type of explanation to another?
Revised: Did you prefer one type of explanation to another?

Original: Was this preference governed by the complexity of the input sentences, etc.?
Revised: Was this preference governed by the complexity of the input sentences, etc.?

Original: https://docs.google.com/forms/u/0/d/1lGmXZZcf4LFlMeUzmN5GALCC1R0FUiOyMVM3Q...
Revised: https://docs.google.com/forms/u/0/d/1lGmXZZcf4LFlMeUzmN5GALCC1R0FUiOyMVM3Q...