Original: Visual correspondence-based explanations improve AI robustness and human-AI team accuracy
Revised: Visual correspondence-based explanations improve AI robustness and human-AI team accuracy

Original: Abstract
Revised: Abstract

Original: Explaining artificial intelligence (AI) predictions is increasingly important and even imperative in many high-stakes applications where humans are the ultimate decision makers.
Revised: Explaining artificial intelligence (AI) predictions is increasingly important and even imperative in many high-stakes applications where humans are the ultimate decision makers.

Original: In this work, we propose two novel architectures of self-interpretable image classifiers that first explain, and then predict (as opposed to post-hoc explanations) by harnessing the visual correspondences between a query image and exemplars.
Revised: In this work, we propose two novel architectures of self-interpretable image classifiers that first explain, and then predict (as opposed to post-hoc explanations) by harnessing the visual correspondences between a query image and exemplars.

Original: Our models consistently improve (+1 to +4 points) on out-of-distribution (OOD) datasets while performing marginally worse (-1 to -2 points) on in-distribution tests than ResNet-50 and a k -nearest neighbor classifier (kNN).
Revised: Our models consistently improve (+1 to +4 points) on out-of-distribution (OOD) datasets while performing marginally worse (-1 to -2 points) on in-distribution tests than ResNet-50 and a k -nearest neighbor classifier (kNN).

Original: Via a large-scale, human study on ImageNet and CUB, our correspondence-based explanations are found to be more useful to users than kNN explanations.
Revised: Via a large-scale, human study on ImageNet and CUB, our correspondence-based explanations are found to be more useful to users than kNN explanations.

Original: Our explanations help users more accurately reject AI’s wrong decisions than all other tested methods.
Revised: Our explanations help users more accurately reject AI’s wrong decisions than all other tested methods.

Original: Interestingly, for the first time, we show that it is possible to achieve complementary human-AI team accuracy (i.e., that is higher than either AI-alone or human-alone), in ImageNet and CUB image classification tasks.
Revised: Interestingly, for the first time, we show that it is possible to achieve complementary human-AI team accuracy (i.e., that is higher than either AI-alone or human-alone), in ImageNet and CUB image classification tasks.

Original:  Introduction
Revised:  Introduction

Original: Comparing the input image with training-set exemplars is the backbone for many applications, such as face identification [29], bird identification [17, 79] and image search [79].
Revised: Comparing the input image with training-set exemplars is the backbone for many applications, such as face identification [29], bird identification [17, 79] and image search [79].

Original: This non-parametric approach may improve classification accuracy on out-of-distribution (OOD) data [29, 75, 79, 57] and enables a class of prototype-based explanations [17, 52, 53, 66, 41] that provide insights into the decision making of Artificial Intelligence (AI) systems.
Revised: This non-parametric approach may improve classification accuracy on out-of-distribution (OOD) data [29, 75, 79, 57] and enables a class of prototype-based explanations [17, 52, 53, 66, 41] that provide insights into the decision making of Artificial Intelligence (AI) systems.

Original: Interestingly, prototype-based explanations are more effective in improving human classification accuracy [55, 24, 42] than attribution maps—a common eXplainable AI (XAI) technique in computer vision.
Revised: Interestingly, prototype-based explanations are more effective in improving human classification accuracy [55, 24, 42] than attribution maps—a common eXplainable AI (XAI) technique in computer vision.

Original: Yet, it remains an open question how to make prototype-based XAI classifiers (1) accurate on in-distribution and OOD data and (2) improve human decisions.
Revised: Yet, it remains an open question how to make prototype-based XAI classifiers (1) accurate on in-distribution and OOD data and (2) improve human decisions.

Original: For example, in face identification, AIs can be confused by partially occluded, never-seen faces and are unable to explain their decisions to users, causing numerous people falsely arrested [5, 7, 3, 6] or wrongly denied unemployment benefits [1] by the law enforcement.
Revised: For example, in face identification, AIs can be confused by partially occluded, never-seen faces and are unable to explain their decisions to users, causing numerous people falsely arrested [5, 7, 3, 6] or wrongly denied unemployment benefits [1] by the law enforcement.

Original: To address the above questions, we propose two interpretable [61], (i.e., first-explain-then-decide)  classifiers that perform three common steps: (1) rank the training-set images based on their distances to the input using image-level features; (2) re-rank the top-50 shortlisted candidates by theirpatchwise correspondences w.r.t. the input [51, 29]; and then (3) take the dominant class among the top-
Revised: To address the above questions, we propose two interpretable [61], (i.e., first-explain-then-decide) image classifiers that perform three common steps: (1) rank the training-set images based on their distances to the input using image-level features; (2) re-rank the top-50 shortlisted candidates by their

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  candidates as the predicted label for the input.
Revised: patch-wise correspondences w.r.t. the input [51, 29]; and then (3) take the dominant class among the top-20 candidates as the predicted label.

Original: That is, our classifiers base their decisions on a set of support image-patch pairs, whichare also presented as explanations to users (Figs. 1b and 2b).
Revised: That is, our classifiers base their decisions on a set of support image-patch pairs, which also serve as explanations to users (Figs. 1b and 2b).

Original: Our main findings include:
Revised: Our main findings include:

Original: • On ImageNet, a simple k -nearest-neighbor classifier (kNN) based on ResNet-50 features slightly but consistently outperforms ResNet-50 on many OOD datasets (Sec. 3.1).
Revised: • On ImageNet, a simple k -nearest-neighbor classifier (kNN) based on ResNet-50 features slightly but consistently outperforms ResNet-50 on many OOD datasets (Sec. 3.1).

Original: This is further improved after a re-ranking step based on patch-wise similarity (Sec. 3.2).
Revised: This is further improved after a re-ranking step based on patch-wise similarity (Sec. 3.2).

Original: • Via a large-scale human study, we find visual correspondence-based explanations to improve AI-assisted, human-alone accuracy and human-AI team accuracy on ImageNet and CUB over the baseline kNN explanations (Sec. 3.4).
Revised: • Via a large-scale human study, we find visual correspondence-based explanations to improve AI-assisted, human-alone accuracy and human-AI team accuracy on ImageNet and CUB over the baseline kNN explanations (Sec. 3.4).

Original: • Having interpretable AIs label images that they are confident and humans label the rest yields better accuracy than letting AIs or humans alone label all images (Sec. 3.5 and Appendix M).
Revised: • Having interpretable AIs label images that they are confident and humans label the rest yields better accuracy than letting AIs or humans alone label all images (Sec. 3.5 and Appendix M).

Original: To the best of our knowledge, our work is the first to demonstrate the utility of correspondence-based explanations to users on ImageNet [63] and CUB [71] classification tasks.
Revised: To the best of our knowledge, our work is the first to demonstrate the utility of correspondence-based explanations to users on ImageNet [63] and CUB [71] classification tasks.

Original:  Methods 2.1 Datasets
Revised:  Methods 2.1 Datasets

Original: We test our ImageNet classifiers on the original 50K-image ILSVRC 2012 ImageNet validation set (i.e., in-distribution data) and four common OOD benchmarks below.
Revised: We test our ImageNet classifiers on the original 50K-image ILSVRC 2012 ImageNet validation set (i.e., in-distribution data) and four common OOD benchmarks below.

Original: ImageNet-R [35] contains 30K images in 200 ImageNet categories, mostly artworks – ranging from cartoons to video game renditions.
Revised: ImageNet-R [35] contains 30K images in 200 ImageNet categories, mostly artworks – ranging from cartoons to video-game renditions.

Original: ImageNet-Sketch [72] consists of 50,889 black-and-white sketches of all 1,000 ImageNet classes.
Revised: ImageNet-Sketch [72] consists of 50,889 black-and-white sketches of all 1,000 ImageNet classes.

Original: DAmageNet [18] consists of 50K ImageNet validation-set images that contain universal, adversarial perturbations for fooling classifiers.
Revised: DAmageNet [18] consists of 50K ImageNet validation-set images that contain universal, adversarial perturbations for fooling classifiers.

Original: Adversarial Patch [15] are 50K ImageNet validation-set images that are modified to contain an adversarial patch that aims to cause ResNet-50 [31] into labeling every image toaster (see Fig. 2).
Revised: Adversarial Patch [15] are 50K ImageNet validation-set images that are modified to contain an adversarial patch that aims to cause ResNet-50 [31] into labeling every image toaster (see Fig. 2).

Original: Using the implementation by [2], we generate this dataset, which causes ResNet-50 accuracy to drop from 76.13% to 55.04% (Table 1).
Revised: Using the implementation by [2], we generate this dataset, which causes ResNet-50 accuracy to drop from 76.13% to 55.04% (Table 1).

Original: See Appendix A.5 for how to download and generate this dataset.
Revised: See Appendix A.5 for how to download and generate this dataset.

Original: CUB-200-2011 [71] (hereafter, CUB) is a fine-grained, bird-image classification task chosen to complement ImageNet.
Revised: CUB-200-2011 [71] (hereafter, CUB) is a fine-grained, bird-image classification task chosen to complement ImageNet.

Original: CUB contains 11,788 images (5,994/5,794 for train/test) of 200 bird species.
Revised: CUB contains 11,788 images (5,994/5,794 for train/test) of 200 bird species.

Original: 2.2 Classifiers
Revised: 2.2 Classifiers

Original: We harness the same ResNet-50 layer4 backbone [8] as the main feature extractor for all four main classifiers, including our two interpretable models.
Revised: We harness the same ResNet-50 layer4 backbone [8] as the main feature extractor for all four main classifiers, including our two interpretable models.

Original: Therefore, to test the effectiveness of our models, we compare them with (1) a vanilla ResNet-50 classifier; and (2) a kNN classifier that uses the same pretrained layer4 features.
Revised: Therefore, to test the effectiveness of our models, we compare them with (1) a vanilla ResNet-50 classifier; and (2) a kNN classifier that uses the same pretrained layer4 features.

Original: We report the top-1 accuracy of all classifiers in Table 1.
Revised: We report the top-1 accuracy of all classifiers in Table 1.

Original: ResNet-50 For experiments on ImageNet and its four OOD benchmarks, we use the ImageNettrained ResNet-50 from TorchVision [8] (top-1 accuracy: 76.13%).
Revised: ResNet-50 For experiments on ImageNet and its four OOD benchmarks, we use the ImageNettrained ResNet-50 from TorchVision [8] (top-1 accuracy: 76.13%).

Original: For CUB, we take the ResNet-50 pretrained on iNaturalist [70] from [53] (hereafter, iNaturalist
Revised: For CUB, we take the ResNet-50 pretrained on iNaturalist [70] from [53] (hereafter, iNaturalist

Original: ResNet) and retrain only the last 200-output classification layer (right after avgpool ) to create a competitive, baseline ResNet-50 classifier for CUB (top-1 accuracy: 85 . 83 %).
Revised: ResNet) and retrain only the last 200-output classification layer (right after avgpool ) to create a competitive, baseline ResNet-50 classifier for CUB (top-1 accuracy: 85 . 83 %).

Original: See Appendix A.1 for finetuning details.
Revised: See Appendix A.1 for finetuning details.

Original: kNN We implement a vanilla kNN classifier that operates at the avgpool of the last convolutional layer of ResNet-50.
Revised: kNN We implement a vanilla kNN classifier that operates at the avgpool of the last convolutional layer of ResNet-50.

Original: That is, given a query image Q , we sort all training-set images { G i } based on their distance D ( Q, G i ) , which is the cosine distance between the two corresponding image features f ( Q ) and f ( G i )
Revised: That is, given a query image Q , we sort all training-set images { G i } based on their distance D ( Q, G i ) , which is the cosine distance between the two corresponding image features f ( Q ) and f ( G i )

Original: ∈
Revised: ∈

Original: R 2048 where f ( . ) outputs the avgpool feature of layer4 (see code) of ResNet-50.
Revised: R 2048 where f ( . ) outputs the avgpool feature of layer4 (see code) of ResNet-50.

Original: The predicted label of Q is the dominant class among the top- k images.
Revised: The predicted label of Q is the dominant class among the top- k nearest neighbors.

Original: We choose k = 20 as it performs the best among the tested values of k ∈ { 10 , 20 , 50 , 100 } .[Figure] a) EMD-Corr: First compute
Revised: We choose k =as it performs the best among the tested values of k ∈ { 10 , 20 , 50 , 100 } .

Original: [Figure][Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure] EMD-Corr As kNN compares images using only image-level features, it lacks the capability of paying attention to fine details in images.
Revised: EMD-Corr As kNN compares images using only image-level features, it lacks the capability of paying attention to fine details in images.

Original: Therefore, we propose EMD-Corr, a visual correspondencebased classifier that (1) re-ranks the top- N (here, N = 50 ) candidates of kNN using their Earth
Revised: Therefore, we propose EMD-Corr, a visual correspondencebased classifier that (1) re-ranks the top- N (here, N = 50 ) candidates of kNN using their Earth

Original: Mover’s Distance (EMD) with the query in a patch embedding space (see Fig. 3a); and (2), similarly to kNN, takes the dominant class among the re-ranked top-20 as the predicted label.
Revised: Mover’s Distance (EMD) with the query in a patch embedding space (see Fig. 3a); and (2), similarly to kNN, takes the dominant class among the re-ranked top-20 as the predicted label.

Original: That is, our PyTorch implementation is the same as that in [79, 29] except for three key differences.
Revised: That is, our PyTorch implementation is the same as that in [79, 29] except for three key differences.

Original: First, using layer4 features (7 × 7 × 2048), we divide an image into 49 patches, whose embeddings are ∈ R 2048 .
Revised: First, using layer4 features (7 × 7 × 2048), we divide an image into 49 patches, whose embeddings are ∈ R 2048 .

Original: Second, for interpretability, in re-ranking, we only use patch-wise EMD instead of a linear combination of image-level cosine distance and patch-level EMD as in [79, 29], which makes it more opaque how a specific image patch contributes to re-ranking.
Revised: Second, for interpretability, in re-ranking, we only use patch-wise EMD instead of a linear combination of image-level cosine distance and patch-level EMD as in [79, 29], which makes it more opaque how a specific image patch contributes to re-ranking.

Original: Third, while the original deep patch-wise EMD [79, 29] between two images (see Eq. 3) is defined as the sum over the weighted cosine distances of all 49 × 49 patch pairs, we only use L = 5 pairs as explanations and therefore, only sum over the corresponding 5 flow weights returned by Sinkhorn optimization [21].
Revised: Third, while the original deep patch-wise EMD [79, 29] between two images (see Eq. 3) is defined as the sum over the weighted cosine distances of all 49 × 49 patch pairs, we only use L = 5 pairs as explanations and therefore, only sum over the corresponding 5 flow weights returned by Sinkhorn optimization [21].

Original: We find N = 50 to perform the best among N ∈ { 50 , 100 , 200 } .
Revised: We find N = 50 to perform the best among N ∈ { 50 , 100 , 200 } .

Original: We choose L = 5 , which is also the most common in nearest-neighbor visualizations [47, 45].
Revised: We choose L = 5 , which is also the most common in nearest-neighbor visualizations [47, 45].

Original: In preliminary experiments, we find
Revised: In preliminary experiments, we find

Original: L = { 9 , 16 , 25 } to yield so dense correspondence visualizations that hurt user interpretation and
Revised: L = { 9 , 16 , 25 } to yield so dense correspondence visualizations that hurt user interpretation and

Original: L = 3 to under-inform users.
Revised: L = 3 to under-inform users.

Original: See Appendix A.3 for more description of EMD-Corr.
Revised: See Appendix A.3 for more description of EMD-Corr.

Original: CHM-Corr EMD-Corr first measures the pair-wise cosine distances for all 49 × 49 pairs of patches, and then computes the EMD flow weights for these pairs (Fig. 3a).
Revised: CHM-Corr EMD-Corr first measures the pair-wise cosine distances for all 49 × 49 pairs of patches, and then computes the EMD flow weights for these pairs (Fig. 3a).

Original: To leverage the recent impressive end-to-end correspondence methods [51, 60, 48], we also propose CHM-Corr (Fig. 3b), a visual correspondence-based classifier that operates in the opposite manner to EMD-Corr.
Revised: To leverage the recent impressive end-to-end correspondence methods [51, 60, 48], we also propose CHM-Corr (Fig. 3b), a visual correspondence-based classifier that operates in the opposite manner to EMD-Corr.

Original: That is, first, we divide the query image Q, into 7 × 7 non-overlapping patches (i.e. as in EMD-Corr) and find one corresponding patch in G i for each of the 49 patches of Q using a state-of-the-art correspondence method (here, CHM [51]).
Revised: That is, first, we divide the query image Q into 7 × 7 non-overlapping patches (i.e. as in EMD-Corr) and find one corresponding patch in G i for each of the 49 patches of Q using a state-of-the-art correspondence method (here, CHM [51]).

Original: Second, for the query Q , we generate a cross-correlation (CC) map (Fig. 3) [29], i.e. a heatmap of cosine similarity scores between the layer4 embeddings of the patches of the query Q and the image-embedding ( avgpool after layer4 ) of each training-set image G i .
Revised: Second, for the query Q , we generate a cross-correlation (CC) map (Fig. 3) [29], i.e. a heatmap of cosine similarity scores between the layer4 embeddings of the patches of the query Q and the image-embedding ( avgpool after layer4 ) of each training-set image G i .

Original: Third, we binarize the heatmap (using the optimal threshold T = 0 .
Revised: Third, we binarize the heatmap (using the optimal threshold T = 0 .

Original: 55 found on a held-out training subset) to identify a set of the most important patches in Q and compute the cosine similarity between each such patch and the corresponding patch in G i (i.e., following the CHM correspondence mappings).
Revised: 55 found on a held-out training subset) to identify a set of the most important patches in Q and compute the cosine similarity between each such patch and the corresponding patch in G i (i.e., following the CHM correspondence mappings).

Original: Finally, the similarity score D ( Q, G i ) in CHM-Corr is the sum over the L = 5 patch pairs of the highest cosine similarity across Q and G i .
Revised: Finally, the similarity score D ( Q, G i ) in CHM-Corr is the sum over the L = 5 patch pairs of the highest cosine similarity across Q and G i .

Original: After testing NC-Net [60], ANC-Net [48], and CHM [51] in our classifier, we choose CHM as it has the fastest runtime and the best accuracy.
Revised: After testing NC-Net [60], ANC-Net [48], and CHM [51] in our classifier, we choose CHM as it has the fastest runtime and the best accuracy.

Original: Unlike ResNet-50 [31], which operates at the 224 ×resolution, CHM uses a ResNet-101 backbone that expects a pair of 240 × 240 images.
Revised: Unlike ResNet-50 [31], which operates at the 224 ×resolution, CHM uses a ResNet-101 backbone that expects a pair of 240 × 240 images.

Original: Therefore, in pre-processing, we resize and center-crop each original ImageNet sample differently according to the input sizes of ResNet-50 and CHM.
Revised: Therefore, in pre-processing, we resize and center-crop each original ImageNet sample differently according to the input sizes of ResNet-50 and CHM.

Original: See Appendix A.4 for more description of CHM-Corr.
Revised: See Appendix A.4 for more description of CHM-Corr.

Original: CHM-Corr+ classifier based on five groundtruth keypoints of birds In EMD-Corr and CHMCorr, we use CC to infer the importance weights of patches.
Revised: CHM-Corr+ classifier based on five groundtruth keypoints of birds In EMD-Corr and CHMCorr, we use CC to infer the importance weights of patches.

Original: To understand the effectiveness of CC in weighting patches, on CUB, we compare our EMD-Corr and CHM-Corr to CHM-Corr+, a CHM-Corr variant where we use a set of five human-defined important patches instead of those inferred by CC.
Revised: To understand the effectiveness of CC in weighting patches, on CUB, we compare our EMD-Corr and CHM-Corr to CHM-Corr+, a CHM-Corr variant where we use a set of five human-defined important patches instead of those inferred by CC.

Original: That is, for each CUB image, instead of taking the five CC-derived important patches (Fig. 3b), we use at most five patches that correspond to a set of five pre-defined keypoints (beak, neck, right wing, right feet, tail), each representing a common body part according to bird identification guides [25] for ornithologists.
Revised: That is, for each CUB image, instead of taking the five CC-derived important patches (Fig. 3b), we use at most five patches that correspond to a set of five pre-defined keypoints (beak, neck, right wing, right feet, tail), each representing a common body part according to bird identification guides [25] for ornithologists.

Original: From the five patches in the query image, we then use CHM to find five corresponding patches in a training-set image, and take the sum of five cosine similarities as the total patch-wise similarity between two images in re-ranking.
Revised: From the five patches in the query image, we then use CHM to find five corresponding patches in a training-set image, and take the sum of five cosine similarities as the total patch-wise similarity between two images in re-ranking.

Original: A query image may have < 5 important patches if some keypoint is occluded.
Revised: A query image may have < 5 important patches if some keypoint is occluded.

Original: That is, evaluating
Revised: That is, evaluating

Original: CHM-Corr+ alone provides an estimate of how hard bird identification on CUB is if the model harnesses five well-known bird features.
Revised: CHM-Corr+ alone provides an estimate of how hard bird identification on CUB is if the model harnesses five well-known bird features.

Original: 2.3 User-study design
Revised: 2.3 User-study design

Original: The interpretable classifiers (Sec. 2.2) are not only capable of classifying images but also generating explanations , which may inform and improve users’ decision-making [55].
Revised: The interpretable classifiers (Sec. 2.2) are not only capable of classifying images but also generating explanations , which may inform and improve users’ decision-making [55].

Original: Here, we design a largescale study to understand the effectiveness of explanations in two human-AI interaction models in classification (see Fig. 4): Model 1: Users make all the decisions after observing the input, AI decisions, and explanations (Fig. 4a).
Revised: Here, we design a largescale study to understand the effectiveness of explanations in two human-AI interaction models in classification (see Fig. 4): Model 1: Users make all the decisions after observing the input, AI decisions, and explanations (Fig. 4a).

Original: Model 2: AIs make decisions on only inputs that they are the most confident, leaving the rest for users to label.
Revised: Model 2: AIs make decisions on only inputs that they are the most confident, leaving the rest for users to label.

Original: That is, model 2 (Fig.
Revised: That is, model 2 (Fig.

Original: 4b–c) is a practical scenario where we offload most inputs to AIs while users only handle the harder inputs.
Revised: 4b–c) is a practical scenario where we offload most inputs to AIs while users only handle the harder cases.

Original: Like [55], we show each user: (1) a query image; (2) AI top-1 label and confidence score; and (3) an explanation (here, available in kNN, EMD-Corr, CHM-Corr, and CHM-Corr+, but not in ResNet-50).
Revised: Like [55], we show each user: (1) a query image; (2) AI top-1 label and confidence score; and (3) an explanation (here, available in kNN, EMD-Corr, CHM-Corr, and CHM-Corr+, but not in ResNet-50).

Original: We ask users to decide Y/N whether the top-1 label is correct (example screen in Fig.
Revised: We ask users to decide Y/N whether the top-1 label is correct (example screen in Fig.

Original: A7).
Revised: A7).

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] whether the AI’s predicted label is correct given explanations.

Original: [Figure]
Revised: model

Original: the is decisions are automatically accepted if AI is highly confident 
Revised: AI decisions are automatically accepted if AI is highly confident (b) .

Original: Otherwise, humans will make decisions (c) in the same fashion as the model 
Revised: Otherwise, humans will make decisions (c) in the same fashion as the model 1.

Original: 2.3.1 Explanation methods
Revised: 2.3.1 Explanation methods

Original: We test the explanations of four main classifiers: ResNet-50, kNN, EMD-Corr, and CHM-Corr.
Revised: We test the explanations of four main classifiers: ResNet-50, kNN, EMD-Corr, and CHM-Corr.

Original: Additionally, we test two ablated versions (i.e., EMD-NN and CHM-NN) of the explanations of
Revised: Additionally, we test two ablated versions (i.e., EMD-NN and CHM-NN) of the explanations of

Original: EMD-Corr and CHM-Corr.
Revised: EMD-Corr and CHM-Corr.

Original: In total, we test 6 explanation methods (see examples in Appendix D).
Revised: In total, we test 6 explanation methods (see examples in Appendix D).

Original: ResNet-50 is a representative black-box classifier, which only outputs a top-1 label and a confidence score but no explanations .
Revised: ResNet-50 is a representative black-box classifier, which only outputs a top-1 label and a confidence score (i.e., no explanations ).

Original: kNN explanations From the top-20 nearest neighbors (as k = 20 in our kNN), we show the first five images that are from the predicted class (example in Fig. 1a).
Revised: kNN explanations From the top-20 nearest neighbors (as k = 20 in our kNN), we show the first five images that are from the predicted class (example in Fig. 1a).

Original: In some cases where the predicted class has only M < 5 exemplars in the top-20, we still show only those M images (see Fig. A44).
Revised: In some cases where the predicted class has only M < 5 exemplars in the top-20, we still show only those M images (see Fig. A46).

Original: That is, we only show at most five neighbors following prior works [55, 67, 49, 65] that reported utility of such few-image explanations.
Revised: That is, we only show at most five neighbors following prior works [55, 67, 49, 65] that reported utility of such few-image explanations.

Original: We find explanations consisting of ≥ 10 images such as those of
Revised: We find explanations consisting of ≥ 10 images such as those of

Original: ProtoPNet [17] are hard to interpret for users [42].
Revised: ProtoPNet [17] are hard to interpret for users [42].

Original: Note that our kNN explanations consist of five support images for each decision of kNN (described in Sec. 2.2) as opposed to the post-hoc nearest examples in [55], which do not reflect a classifier’s decisions.
Revised: Note that our kNN explanations consist of five support images for each decision of kNN (described in Sec. 2.2) as opposed to the post-hoc nearest examples in [55], which do not reflect a classifier’s decisions.

Original: EMD-Corr and CHM-Corr explanations
Revised: EMD-Corr and CHM-Corr explanations

Original: As EMD-Corr and CHM-Corr re-rank the topcandidates (shortlisted by kNN) and take the dominant class among the resultant top-20 as the predicted label, we show the five nearest neighbors from the predicted class as in kNN explanations.
Revised: As EMD-Corr and CHM-Corr re-rank the topcandidates (shortlisted by kNN) and take the dominant class among the resultant top-20 as the predicted label, we show the five nearest neighbors from the predicted class as in kNN explanations.

Original: Instead of showing only five post-reranking neighbors, we also annotate, in each image, all five patches (example in Fig.
Revised: Instead of showing only five post-reranking neighbors, we also annotate, in each image, all five patches (example in Fig.

Original: 1b and Fig. 2b) that contribute to the patch-wise re-ranking (Sec. 2.2).
Revised: 1b and Fig. 2b) that contribute to the patch-wise re-ranking (Sec. 2.2).

Original: EMD-NN and CHM-NN
Revised: EMD-NN and CHM-NN

Original: To understand the effects of showing correspondences in the explanations to EMD-Corr and CHM-Corr users, we also test an ablated version where we show exactly the same explanations but without the patch annotations (bottom panels in Fig.
Revised: To understand the effects of showing correspondences in the explanations to EMD-Corr and CHM-Corr users, we also test an ablated version where we show exactly the same explanations but without the patch annotations (bottom panels in Fig.

Original: 1a and Fig.
Revised: 1a and Fig.

Original: 2a).
Revised: 2a).

Original: Confidence scores While ResNet-50’s confidence score is the top-1 output softmax probability, the confidence of kNN, EMD-Corr, and CHM-Corr is the count of the predicted-class examples among the top k = 20.
Revised: Confidence scores While ResNet-50’s confidence score is the top-1 output softmax probability, the confidence of kNN, EMD-Corr, and CHM-Corr is the count of the predicted-class examples among the top k = 20.

Original: In human studies, we display this confidence in percentage (e.g. 10% instead of 2/20;
Revised: In human studies, we display this confidence in percentage (e.g. 10% instead of 2/20;

Original: Fig.
Revised: Fig.

Original: A44) to be consistent with the confidence score of ResNet-50.
Revised: A46) to be consistent with the confidence score of ResNet-50.

Original: Examples of all 6 explanation methods are in Appendix D.

Original: 2.3.2 ImageNet and CUB datasets
Revised: 2.3.2 ImageNet and CUB datasets

Original: For XAI evaluation, we run two human studies, one on ImageNet and one on CUB.
Revised: For XAI evaluation, we run two human studies, one on ImageNet and one on CUB.

Original: For ImageNet, we use ImageNet-ReaL [14] labels in attempt to minimize the confounders of the human evaluation as ImageNet labels are sometimes misleading and inaccurate to users [55].
Revised: For ImageNet, we use ImageNet-ReaL [14] labels in attempt to minimize the confounders of the human evaluation as ImageNet labels are sometimes misleading and inaccurate to users [55].

Original: Nearest-neighbor images To generate the nearest-neighbor explanations for kNN, EMD-Corr, and
Revised: Nearest-neighbor images To generate the nearest-neighbor explanations for kNN, EMD-Corr, and

Original: CHM-Corr, we search for neighbors in the entire training set of ImageNet or CUB (no filtering).
Revised: CHM-Corr, we search for neighbors in the entire training set of ImageNet or CUB (no filtering).

Original: Query images In attempt to ensure the quality of the query images that we ask users to label, from 50K-image ImageNet validation set, we discard images that: (a) do not have an ImageNet ReaL [14] label; (b) are grayscale or low-resolution (i.e., either width or height < 224 px) as in [55]; (c) have duplicates in the ImageNet training set (see Appendix L), resulting in 44,424 images available for sampling for the study.
Revised: Query images In attempt to ensure the quality of the query images that we ask users to label, from 50K-image ImageNet validation set, we discard images that: (a) do not have an ImageNet-ReaL [14] label; (b) are grayscale or low-resolution (i.e., either width or height < 224 px) as in [55]; (c) have duplicates in the ImageNet training set (see Appendix L), resulting in 44,424 images available for sampling for the study.

Original: In CUB, we sample from the entire 5,794-image test set and apply no filters.
Revised: In CUB, we sample from the entire 5,794-image test set and apply no filters.

Original: 2.3.3 Training, Validation, and Test phases
Revised: 2.3.3 Training, Validation, and Test phases

Original: From the set of query images (Sec. 2.3.2), we sample images for three phases in a user study: Training, validation, and test.
Revised: From the set of query images (Sec. 2.3.2), we sample images for three phases in a user study: Training, validation, and test.

Original: Following [55], we first introduce participants to the task and provide themtraining examples.
Revised: Following [55], we first introduce participants to the task and provide themtraining examples.

Original: Then, each user is given a validation job (10 trials for ImageNet and 5 for CUB), where they must score 100% in order to be invited to our  test phase of 30 trials.
Revised: Then, each user is given a validation job (10 trials for ImageNet and 5 for CUB), where they must score 100% in order to be invited to our 30-trial test phase.

Original: Otherwise, they will be rejected and unpaid.
Revised: Otherwise, they will be rejected and unpaid.

Original: Among the 10 validation trials for ImageNet, 5 are correctly-labeled andare misclassified by AIs.
Revised: Among the 10 validation trials for ImageNet, 5 are correctly-labeled and 5 are misclassified by AIs.

Original: This ratio is 3/2 for CUB validation (examples in Appendix E.2).
Revised: This ratio is 3/2 for CUB validation (examples in Appendix E.2).

Original: Right before each trial, we describe the  top-1 class to users by showing  3 training-set images and a 1-sentence WordNet description for each ImageNet class.
Revised: Right before each trial, we describe the AI’s top-1 label to users by showing them 3 training-set images and a 1-sentence WordNet description for each ImageNet class.

Original: For CUB classes, we show 6 representative images (instead of 3) for users to better recognize the characteristics of each bird (see Fig.
Revised: For CUB classes, we showrepresentative images (instead of 3) for users to better recognize the characteristics of each bird (see Fig.

Original: A6).
Revised: A6).

Original: Sampling For every classifier, we randomly sample 300 correctly and 300 incorrectly predicted images together with their corresponding explanations for the test trials.
Revised: Sampling For every classifier, we randomly sample 300 correctly- and 300 incorrectly-predicted images together with their corresponding explanations for the test trials.

Original: Over all 6 explanation methods, we have 2 datasets × 600 images × 6 methods = 7,200 test images in total.
Revised: Over all 6 explanation methods, we have 2 datasets × 600 images × 6 methods = 7,200 test images in total.

Original: 2.3.4 Participants
Revised: 2.3.4 Participants

Original: We host human studies on Gorilla [11] and recruit lay participants who are native English speakers worldwide via Prolific [56] at a pay rate of USD 13.5 / hr.
Revised: We host human studies on Gorilla [11] and recruit lay participants who are native English speakers worldwide via Prolific [56] at a pay rate of USD 13.5 / hr.

Original: We have 360 and 355 users who successfully passed our validation test for ImageNet and CUB datasets, respectively.
Revised: We have 360 and 355 users who successfully passed our validation test for ImageNet and CUB datasets, respectively.

Original: We remove low-quality, bottom-outlier submissions, i.e., who score ≤ 0 .
Revised: We remove low-quality, bottom-outlier submissions, i.e., who score ≤ 0 .

Original: 55 (nearly random chance), resulting in 354 andsubmissions for ImageNet and CUB, respectively.
Revised: 55 (near-random accuracy), resulting in 354 andsubmissions for ImageNet and CUB, respectively.

Original: In each dataset, every explanation method is tested on ∼ 60 users and each pair of (query, explanation) is seen by almost 3 users (details in Table. 2).
Revised: In each dataset, every explanation method is tested on ∼ 60 users and each pair of (query, explanation) is seen by almost 3 users (details in Table 2).

Original:  Experimental Results
Revised:  Experimental Results

Original: 3.1 ImageNet kNN classifiers improve upon ResNet-50 on out-of-distribution datasets
Revised: 3.1 ImageNet kNN classifiers improve upon ResNet-50 on out-of-distribution datasets

Original: Despite impressive test-set performance, ImageNet-trained convolutional neural networks (CNNs) may fail to generalize to natural OOD data [72] or inputs specifically crafted to fool them [54, 18, 15].
Revised: Despite impressive test-set performance, ImageNet-trained convolutional neural networks (CNNs) may fail to generalize to natural OOD data [72] or inputs specifically crafted to fool them [54, 18, 15].

Original: It is unknown whether prototype-based classifiers can leverage the known exemplars (i.e. support images) to generalize better to unseen, rare inputs.
Revised: It is unknown whether prototype-based classifiers can leverage the known exemplars (i.e. support images) to generalize better to unseen, rare inputs.

Original: To test this question, here, we compare kNN with the baseline ResNet-50 classifier (both described in Sec. 2) on ImageNet and related OOD datasets.
Revised: To test this question, here, we compare kNN with the baseline ResNet-50 classifier (both described in Sec. 2) on ImageNet and related OOD datasets.

Original: On ImageNet and ImageNet-ReaL, kNN performs slightly worse than ResNet-50 byan absolute decrease of -1.36 and -0.99 points, respectively (Table 1).
Revised: On ImageNet and ImageNet-ReaL, kNN performs slightly worse than ResNet-50 by -1.36 and -0. points, respectively (Table 1).

Original: However, interestingly, on all four OOD datasets, kNN consistently outperforms ResNet-50.
Revised: Yet, interestingly, on all four OOD datasets, kNN consistently outperforms ResNet-50.

Original: Notably, kNN improves upon ResNetby +1.66 and +4.26 points on DAmageNet and Adversarial Patch.
Revised: Notably, kNN improves upon ResNet-50 by +1.66 and +4.26 points on DAmageNet and Adversarial Patch.

Original: That is, while ResNet-50 and kNN share the exact same backbone, the kNN’s process of comparing the input image against the training-set examples prove to be beneficial for generalizing to OOD inputs.
Revised: That is, while ResNet-50 and kNN share the exact same backbone, the kNN’s process of comparing the input image against the training-set examples prove to be beneficial for generalizing to OOD inputs.

Original: Intuitively, given the same ImageNet pre-trained feature extractor, it is useful to “look back” at the training-set exemplarsagain to decide a label for hard, long-tail or OOD samples.
Revised: Intuitively, our results suggest that it is useful to “look back” at the training-set exemplars to decide a label for hard, long-tail or OOD images.

Original: Consistently, using the same CUB-finetuned backbone, kNN is only marginally worse than ResNeton CUB (85.46% vs. 85.83%; Table 1).
Revised: Consistently, using the same CUB-finetuned backbone, kNN is only marginally worse than ResNeton CUB (85.46% vs. 85.83%; Table 1).

Original: 3.2 Visual correspondence-based explanations improve kNN robustness further
Revised: 3.2 Visual correspondence-based explanations improve kNN robustness further

Original: Recent work found that re-ranking kNN’s shortlisted candidates using the patch-wise similarity between the query and training set examples can further improve classification accuracy on OOD data for some image matching tasks [29, 79, 75] such as face identification [29].
Revised: Recent work found that re-ranking kNN’s shortlisted candidates using the patch-wise similarity between the query and training set examples can further improve classification accuracy on OOD data for some image matching tasks [29, 79, 75] such as face identification [29].

Original: Furthermore, patch-level comparison is also useful in prototype-based bird classifiers [17, 22].
Revised: Furthermore, patch-level comparison is also useful in prototype-based bird classifiers [17, 22].

Original: Inspired by these prior successes and the fact that EMD-Corr and CHM-Corr base the patch-wise similarity of two images on only 
Revised: Inspired by these prior successes and the fact that EMD-Corr and CHM-Corr base the patch-wise similarity of two images on only 5 patch pairs instead of all 49 × 49 = 2,401 pairs as in [29, 79, 75], here we test whether our two proposed re-rankers are able to improve the test-set accuracy and robustness over kNN.

Original: [Table]
Revised: [Table]

Original: pairs of patches instead of all 49 × 49 = 2,401 pairs as in [29, 79, 75], here we test whether our two proposed re-rankers are able to improve the test-set accuracy and robustness over kNN.

Original: Experiment We run EMD-Corr and CHM-Corr on all datasets and compare their results with that of kNN (Table 1).
Revised: Experiment We run EMD-Corr and CHM-Corr on all datasets and compare their results with that of kNN (Table 1).

Original: Both methods (described in Sec. 2) re-rank the top N = 50 shortlisted candidates returned by kNN and then take the dominant class in the top- k (where k = 20) as the predicted label.
Revised: Both methods (described in Sec. 2) re-rank the top N = 50 shortlisted candidates returned by kNN and then take the dominant class in the top- k (where k = 20) as the predicted label.

Original: ImageNet results Interestingly, despite using only 5 pairs of patches to compute image similarity for re-ranking, both classifiers consistently improve upon kNN further, especially in all OOD datasets.
Revised: ImageNet results Interestingly, despite using only 5 pairs of patches to compute image similarity for re-ranking, both classifiers consistently improve upon kNN further, especially on all OOD datasets.

Original: Overall, EMD-Corr and CHM-Corr outperform kNN and ResNet-50 baselines from +1.27 to +4.
Revised: Overall, EMD-Corr and CHM-Corr outperform kNN and ResNet-50 baselines from +1.27 to +4.

Original: points (Table 1).
Revised: points (Table 1).

Original: Intuitively, in some hard cases where the main object is small, the two Corr classifiers ignore irrelevant patches (e.g. the sky in ibex images; Fig. 1) and only use the five most relevant patches to make decisions.
Revised: Intuitively, in some hard cases where the main object is small, the two Corr classifiers ignore irrelevant patches (e.g. the sky in ibex images; Fig. 1) and only use the five most relevant patches to make decisions.

Original: Similarly, on Adversarial Patch, relying on a few key patches while ignoring adversarial patches enables our classifiers to outperform baselines (Fig. 2).
Revised: Similarly, on Adversarial Patch, relying on a few key patches while ignoring adversarial patches enables our classifiers to outperform baselines (Fig. 2).

Original: See Appendix I for many qualitative examples comparing Corr and kNN predictions.
Revised: See Appendix I for many qualitative examples comparing Corr and kNN predictions.

Original: CUB results Interestingly, using the same ImageNet-pretrained backbones, EMD-Corr outperforms kNN by an absolute +5.57 points when tested on CUB (60.29% vs. 54.72%; Table 1).
Revised: CUB results Interestingly, using the same ImageNet-pretrained backbones, EMD-Corr outperforms kNN by an absolute +5.57 points when tested on CUB (60.29% vs. 54.72%; Table 1).

Original: However, this difference vanishes when using CUB-pretrained backbones (Table 1; 84.98% vs. 85.46%).
Revised: However, this difference vanishes when using CUB-pretrained backbones (Table 1; 84.98% vs. 85.46%).

Original: Our CUB and ImageNet results are consistent and together reveal a trend: On i.i.d test sets, Corr models perform on par with kNN; however, on OOD images, they consistently outperform kNN, highlighting the benefits of patch-wise comparison.
Revised: Our CUB and ImageNet results are consistent and together reveal a trend: On i.i.d test sets, Corr models perform on par with kNN; however, on OOD images, they consistently outperform kNN, highlighting the benefits of patch-wise comparison.

Original: 3.3 Corr classifiers 
Revised: 3.3 Corr classifiers leverage five patches that are more important than five bird keypoints

Original: leverage five patches that are more important than five bird keypoints

Original: EMD-Corr and CHM-Corr harness five patches per image for computing a patch-wise similarity score for a pair of images (Sec. 2.2)
Revised: EMD-Corr and CHM-Corr harness five patches per image for computing a patch-wise similarity score for a pair of images (Sec. 2.2)

Original: As these five patches are automatically inferred by cross-correlation (Fig. 3), it is interesting to understand further whether replacing these five patches by five user-defined patches  would improve classification accuracy.
Revised: As these five patches are automatically inferred by cross-correlation (Fig. 3), it is interesting to understand further whether replacing these five patches by five user-defined patches in [71] would improve classification accuracy.

Original: Experiment Since there are no keypoints provided for ImageNet, we test the importance of the five key patches chosen by Corr methods on CUB because CUB provides ornithologist-defined annotations for each bird image.
Revised: Experiment Since there are no keypoints provided for ImageNet, we test the importance of the five key patches chosen by Corr methods on CUB because CUB provides ornithologist-defined annotations for each bird image.

Original: That is, we create a baseline CHM-Corr+, which is the same as
Revised: That is, we create a baseline CHM-Corr+, which is the same as

Original: CHM-Corr, except that we use five important patches that correspond to five keypoints in a bird image—beak, belly, tail, right wing, and right foot—as described in Sec.
Revised: CHM-Corr, except that we use five important patches that correspond to five keypoints in a bird image—beak, belly, tail, right wing, and right foot—as described in Sec.

Original: We also test CHM-Corr+ sweeping across the number of keypoints ∈ { 5 , 10 , 15 } .
Revised: We also test CHM-Corr+ sweeping across the number of keypoints ∈ { 5 , 10 , 15 } .

Original: Results On CUB, CHM-Corr outperforms CHM-Corr+ despite the fact that the baseline method leverages five human-defined bird keypoints (Table 1; 83.27% vs. 81.51%) while CHM-Corr may also use background patches.
Revised: Results On CUB, CHM-Corr outperforms CHM-Corr+ despite the fact that the baseline method leverages five human-defined bird keypoints (Table 1; 83.27% vs. 81.51%) while CHM-Corr may also use background patches.

Original: Interestingly, when increasing the number of keypoints to 10 and 15, the accuracy of CHM-Corr+ is still lower than that of CHM-Corr (i.e., from 81.51% to 82.34% and 82.27%, respectively).
Revised: Interestingly, when increasing the number of keypoints to 10 and 15, the accuracy of CHM-Corr+ is still lower than that of CHM-Corr (i.e., from 81.51% to 82.34% and 82.27%, respectively).

Original: That is, 15 keypoints may correspond to ≤ 15 different patches per image ( if each keypoint lies in a unique, non-overlapping patch among all the 49 patches per image).
Revised: That is, 15 keypoints may correspond to ≤ 15 different patches per image ( if each keypoint lies in a unique, non-overlapping patch among all the 49 patches per image).

Original: Our results show strong evidence that the five key patches inferred by CC used in EMD- and CHMCorrare more important than bird keypoints and do not necessarily cover the birds.
Revised: Our results show strong evidence that the five key patches inferred by CC used in EMD- and CHMCorr do not necessarily cover the birds but are more important than expert-defined bird keypoints.

Original: Qualitative comparisons between CHM-Corr and CHM-Corr+ predictions are in Appendix J.
Revised: Qualitative comparisons between CHM-Corr and CHM-Corr+ predictions are in Appendix J.

Original: 3.4 On ImageNet-ReaL, correspondence-based explanations are more useful to users than kNN explanations
Revised: 3.4 On ImageNet-ReaL, correspondence-based explanations are more useful to users than kNN explanations

Original: Given that EMD-Corr and CHM-Corr classifiers outperform kNN classifiers on OOD datasets (Sec. 3.2), it is interesting to test how their respective explanations help humans perform classification on ImageNet.
Revised: Given that EMD-Corr and CHM-Corr classifiers outperform kNN classifiers on OOD datasets (Sec. 3.2), it is interesting to test how their respective explanations help humans perform classification on ImageNet.

Original: Furthermore, in image classification, kNN explanations were found to be more useful to humans than saliency maps [55].
Revised: Furthermore, in image classification, kNN explanations were found to be more useful to humans than saliency maps [55].

Original: [Figure]
Revised: [Figure]

Original: e s N e t
Revised: e s N e t

Original: [Figure]
Revised: [Figure]

Original: 5 0 k N N E M D
Revised: 5 0 k N N E M D

Original: [Figure]
Revised: [Figure]

Original: N N E M D
Revised: N N E M D

Original: [Figure]
Revised: [Figure]

Original: C o rr C H M
Revised: C o rr C H M

Original: [Figure]
Revised: [Figure]

Original: N N C H
Revised: N N C H

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Experiment We perform a human study to assess the ImageNet-ReaL classification accuracy of users of each classifier (described in Sec. 2.2) when they are provided with a classifier’s predictions and explanations.
Revised: Experiment We perform a human study to assess the ImageNet-ReaL classification accuracy of users of each classifier (described in Sec. 2.2) when they are provided with a classifier’s predictions and explanations.

Original: That is, we measure the AI-assisted classification accuracy of users following human-AI interaction model 1 (Fig. 4a).
Revised: That is, we measure the AI-assisted classification accuracy of users following human-AI interaction model 1 (Fig. 4a).

Original: We compare the accuracy between user groups of four classifiers ResNet-50, kNN,
Revised: We compare the accuracy between user groups of four classifiers ResNet-50, kNN,

Original: EMD-Corr, and CHM-Corr (described in Sec. 2.3.1).
Revised: EMD-Corr, and CHM-Corr (described in Sec. 2.3.1).

Original: Additionally, to thoroughly assess the impact of showing the correspondence boxes compared to showing only nearest neighbor images (e.g.,
Revised: Additionally, to thoroughly assess the impact of showing the correspondence boxes compared to showing only nearest neighbor images (e.g.,

Original: CHM-Corr vs. CHM-NN in Fig. 1), we test two more user groups of
Revised: CHM-Corr vs. CHM-NN in Fig. 1), we test two more user groups of

Original: EMD-NN and CHM-NN, i.e. the same explanations as those of the
Revised: EMD-NN and CHM-NN, i.e. the same explanations as those of the

Original: Corr classifiers but with the correspondence boxes hidden .
Revised: Corr classifiers but with the correspondence boxes hidden .

Original: Results First, the mean accuracy of kNN users is consistently lower than that of the other models’ users (e.g., 75.76% vs. 78.87% of EMDCorr; Table 2).
Revised: Results First, the mean accuracy of kNN users is consistently lower than that of the other models’ users (e.g., 75.76% vs. 78.87% of EMDCorr; Table 2).

Original: The EMD-Corr improvement over kNN is statistically significant ( p < 0.01 via Mann-Whitney U test; Fig.
Revised: The EMD-Corr improvement over kNN is statistically significant ( p < 0.01 via Mann-Whitney U test; Fig.

Original: 5a)
Revised: 5a)

Original: Second, interestingly, we find the differences between EMD-, CHMCorr and their respective baselines are small and not statistically significant (Fig. 5a).
Revised: Second, interestingly, we find the differences between EMD-, CHMCorr and their respective baselines are small and not statistically significant (Fig. 5a).

Original: That is, on ImageNet-ReaL, quantitatively, showing the correspondence boxes on top of nearest neighbors is not more useful to users.
Revised: That is, on ImageNet-ReaL, quantitatively, showing the correspondence boxes on top of nearest neighbors is not more useful to users.

Original: Third, surprisingly, the users of ResNet-50 (mean accuracy of 81.56%; Table 2) outperform all other methods’ users, suggesting that on ImageNet, a task of many familiar classes to users, ante-hoc explanations hurt user accuracy rather than help.
Revised: Third, surprisingly, the users of ResNet-50 (mean accuracy of 81.56%; Table 2) outperform all other methods’ users, suggesting that on ImageNet, a task of many familiar classes to users, ante-hoc explanations hurt user accuracy rather than help.

Original: Note that in Nguyen et al.
Revised: Note that in Nguyen et al.

Original: [55], post-hoc kNN explanations were found useful to humans compared to not showing any explanations.
Revised: [55], post-hoc kNN explanations were found useful to humans compared to not showing any explanations.

Original: Yet, here, each classifier’s users are provided with a different set of images and AI decisions, which can also influence the user accuracy.
Revised: Yet, here, each classifier’s users are provided with a different set of images and AI decisions, which can also influence the user accuracy.

Original: When ResNet-50 is wrong, their users are substantially better in detecting such misclassifications compared to other models’ users (Fig. A11a).
Revised: When ResNet-50 is wrong, their users are substantially better in detecting such misclassifications compared to other models’ users (Fig. A11a).

Original: 3.5 On CUB fine-grained bird classification, correspondence-based explanations are the most useful to users, helping them to more accurately reject AI misclassifications
Revised: 3.5 On CUB fine-grained bird classification, correspondence-based explanations are the most useful to users, helping them to more accurately reject AI misclassifications

Original: To assess whether the findings on ImageNet-ReaL in Sec.
Revised: To assess whether the findings on ImageNet-ReaL in Sec.

Original: 3.4 generalize to a fine-grained classification task, we repeat the user study on CUB—which is considered much harder to lay users than ImageNet.
Revised: 3.4 generalize to a fine-grained classification task, we repeat the user study on CUB—which is considered much harder to lay users than ImageNet.

Original: Results Interestingly, we find EMD-Corr and CHM-Corr users consistently outperform ResNet-50, kNN, EMD-NN, and CHM-NN users (Table 2).
Revised: Results Interestingly, we find EMD-Corr and CHM-Corr users consistently outperform ResNet-50, kNN, EMD-NN, and CHM-NN users (Table 2).

Original: The differences between EMD-Corr (or CHM-Corr) and every other baseline are statistically significant ( p < 0.05 via Mann-Whitney U test; Fig. 5b).
Revised: The differences between EMD-Corr (or CHM-Corr) and every other baseline are statistically significant ( p < 0.05 via Mann-Whitney U test; Fig. 5b).

Original: That is, on CUB, the visual correspondence boxes help users make more accurate decisions compared to (a) having no explanations at all (ResNet-50); (b) showing nearest neighbors sorted by image similarity only, not patch correspondences (Fig.
Revised: That is, on CUB, the visual correspondence boxes help users make more accurate decisions compared to (a) having no explanations at all (ResNet-50); (b) showing nearest neighbors sorted by image similarity only, not patch correspondences (Fig.

Original: 1a; kNN); and (c) having patch-wise correspondence neighbors but not displaying the boxes (Fig.
Revised: 1a; kNN); and (c) having patch-wise correspondence neighbors but not displaying the boxes (Fig.

Original: A37; CHM-NN and EMD-NN).
Revised: A39; CHM-NN and EMD-NN).

Original: Corr explanations help users reject AI misclassifications while kNN is poorly trust-calibrated
Revised: Corr explanations help users reject AI misclassifications while kNN is poorly trust-calibrated

Original: In an attempt to understand why the two Corr classifiers help users the most, we find that EMD-Corr and CHM-Corr users reject AI predictions at the highest rates (32.70% and 33.73%; Table A6) while kNN users reject the least (18.47%).
Revised: In an attempt to understand why the two Corr classifiers help users the most, we find that EMD-Corr and CHM-Corr users reject AI predictions at the highest rates (32.70% and 33.73%; Table A6) while kNN users reject the least (18.47%).

Original: This might have led to the substantially higher accuracy of CHM-Corr users, compared to all other models’ user groups, when AI predictions are wrong (Fig. A11b; e.g., 53.45% of CHM-Corr vs.
Revised: This might have led to the substantially higher accuracy of CHM-Corr users, compared to all other models’ user groups, when AI predictions are wrong (Fig. A11b; e.g., 53.45% of CHM-Corr vs.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 41.22% of ResNet-50).
Revised: 41.22% of ResNet-50).

Original: That is
Revised: That is

Original: [Table]
Revised: [Table]

Original: CHM-Corr users correctly reject 53.45% of the images that
Revised: CHM-Corr users correctly reject 53.45% of the images that

Original: the
Revised: the

Original: CHM-Corr classifier mislabels.
Revised: CHM-Corr classifier mislabels.

Original: In contrast, kNN users reject the least, only 33.22% of incorrect predictions (Fig. A11b).
Revised: In contrast, kNN users reject the least, only 33.22% of incorrect predictions (Fig. A11b).

Original: kNN explanations tend to fool users into trusting the kNN’s wrong decisions (Fig.
Revised: kNN explanations tend to fool users into trusting the kNN’s wrong decisions (Fig.

Original: 6)—the accuracy of kNN users is 33.22%, much lower than the 41.22% of ResNet-50 users who observe no explanations.
Revised: 6)—the accuracy of kNN users is 33.22%, much lower than the 41.22% of ResNet-50 users who observe no explanations.

Original: On ImageNet (Fig. A11), kNN is also poorly “trust-calibrated” [69, 74].
Revised: On ImageNet (Fig. A11), kNN is also poorly “trust-calibrated” [69, 74].

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We hypothesize that kNN explanations tend to fool users more as their nearest neighbors, by design, show images that are image-wise similar to the query (regardless of whether the kNN prediction is correct or not) while EMD-Corr and CHM-Corr re-rank the images based on patch-wise similarity.
Revised: We hypothesize that kNN explanations tend to fool users more as their nearest neighbors, by design, show images that are image-wise similar to the query (regardless of whether the kNN prediction is correct or not) while EMD-Corr and CHM-Corr re-rank the images based on patch-wise similarity.

Original: Furthermore, we find the images in kNN explanations are also less diverse than those in Corr explanations in both LPIPS [77] and MS-SSIM [73] (Appendix H.2).
Revised: Furthermore, we find the images in kNN explanations are also less diverse than those in Corr explanations in both LPIPS [77] and MS-SSIM [73] (Appendix H.2).

Original: Corr explanations tend to include more diverse images (Fig.
Revised: Corr explanations tend to include more diverse images (Fig.

Original: 6a; top vs. bottom) and provide users with more contrastive evidence in order to reject AI’s incorrect predictions.
Revised: 6a; top vs. bottom) and provide users with more contrastive evidence in order to reject AI’s incorrect predictions.

Original: Additionally, we also hypothesize that users are less confident about AI’s decisions (and thus reject more) when Corr explanations show some background and uninformative patches used in the matching process (e.g., the 1st and 4th image in Fig. 6b).
Revised: Additionally, we also hypothesize that users are less confident about AI’s decisions (and thus reject more) when Corr explanations show some background and uninformative patches used in the matching process (e.g., the 1st and 4th image in Fig. 6b).

Original: Yet, such boxes are not available in kNN explanations.
Revised: Yet, such boxes are not available in kNN explanations.

Original: When Corr explanations allow for more disagreement between AI and users, humans also tend to incorrectly reject AI’s correct predictions more often (Fig. A11b; Corr users are the least accurate among 6 methods when the AI is correct).
Revised: When Corr explanations allow for more disagreement between AI and users, humans also tend to incorrectly reject AI’s correct predictions more often (Fig. A11b; Corr users are the least accurate among 6 methods when the AI is correct).

Original: EMD-Corr and CHM-Corr users score ∼ 4 points below ResNet-50 users (84.94% and 85.99% vs. 89.87% of ResNet-50).
Revised: EMD-Corr and CHM-Corr users score ∼ 4 points below ResNet-50 users (84.94% and 85.99% vs. 89.87% of ResNet-50).

Original: When most users reject AI’s correct predictions, we observe that some discriminative features (e.g., the belly stripes of the Field
Revised: When most users reject AI’s correct predictions, we observe that some discriminative features (e.g., the belly stripes of the Field

Original: Sparrow; Fig. A33c) are often occluded in the query, leading to human-AI disagreement.
Revised: Sparrow; Fig. A35c) are often occluded in the query, leading to human-AI disagreement.

Original:  Related Work
Revised:  Related Work

Original: Patch-wise similarity Calculating patch-wise similarity, either intra-image [23] or inter-image [29, 79, 75], has been useful in many tasks as the comparison enables machines to attend to finegrained details and compute more accurate decisions.
Revised: Patch-wise similarity Calculating patch-wise similarity, either intra-image [23] or inter-image [29, 79, 75], has been useful in many tasks as the comparison enables machines to attend to finegrained details and compute more accurate decisions.

Original: Our EMD-Corr harnesses a similar approach to that in [29, 79]; which, however, was not tested on ImageNet classification as in our work.
Revised: Our EMD-Corr harnesses a similar approach to that in [29, 79]; which, however, was not tested on ImageNet classification as in our work.

Original: Furthermore, we only compute the total patch-wise similarity over the top-5 patch pairs between the query and each exemplar instead of all pairs as in [29, 79].
Revised: Furthermore, we only compute the total patch-wise similarity over the top-5 patch pairs between the query and each exemplar instead of all pairs as in [29, 79].

Original: Compared to recent patch-wise similarity works that use either cross-attention in ViTs [23] or EMD [79, 75, 29, 43], our work is the first to perform human evaluation of the correspondence-based explanations.
Revised: Compared to recent patch-wise similarity works that use either cross-attention in ViTs [23] or EMD [79, 75, 29, 43], our work is the first to perform human evaluation of the correspondence-based explanations.

Original: Prototype-based XAI methods Our work is motivated by the recent finding that exemplar-based explanations are more effective than heatmap-based explanations in improving human classification accuracy [55, 38, 42, 24].
Revised: Prototype-based XAI methods Our work is motivated by the recent finding that exemplar-based explanations are more effective than heatmap-based explanations in improving human classification accuracy [55, 38, 42, 24].

Original: However, showing an entire image as an exemplar without any further localization may be confusing as it is unknown which parts of the image the AI is paying attention to [55, 38].
Revised: However, showing an entire image as an exemplar without any further localization may be confusing as it is unknown which parts of the image the AI is paying attention to [55, 38].

Original: Our EMD-Corr and CHM-Corr present a novel combination of heatmap-based and prototype-based XAI approaches.
Revised: Our EMD-Corr and CHM-Corr present a novel combination of heatmap-based and prototype-based XAI approaches.

Original: None of the prior prototype-based XAI methods that operate at the patch level [17, 53, 79, 22] (see Table 3 in [22]) were tested on humans yet.
Revised: None of the prior prototype-based XAI methods that operate at the patch level [17, 53, 79, 22] (see Table 3 in [22]) were tested on humans yet.

Original: Also, in preliminary tests, we find their explanation formats too dense (i.e., showing over 10 prototypes [17], 9 correspondence pairs per image [22], or an entire prototype tree to humans [52, 53]) to be useful for lay users.
Revised: Also, in preliminary tests, we find their explanation formats too dense (i.e., showing over 10 prototypes [17], 9 correspondence pairs per image [22], or an entire prototype tree to humans [52, 53]) to be useful for lay users.

Original: Another major difference is that our Corr classifiers are nonparametric, allowing the training set to be adjusted or swapped with any external knowledgebase for debugging purposes.
Revised: Another major difference is that our Corr classifiers are nonparametric, allowing the training set to be adjusted or swapped with any external knowledgebase for debugging purposes.

Original: In contrast, recent prototype-based classifiers [17, 53, 79, 22] are parametric, using a set of learned prototypes and thus may not perform well on OOD datasets as EMD-Corr and CHM-Corr.
Revised: In contrast, recent prototype-based classifiers [17, 53, 79, 22] are parametric, using a set of learned prototypes and thus may not perform well on OOD datasets as EMD-Corr and CHM-Corr.

Original: Post-hoc prototype-based explanations Some prototype-based methods are post-hoc [41, 55, 20], i.e., generating explanations to explain a decision after-the-fact, which could be highly unfaithful [61, 62].
Revised: Post-hoc prototype-based explanations Some prototype-based methods are post-hoc [41, 55, 20], i.e., generating explanations to explain a decision after-the-fact, which could be highly unfaithful [61, 62].

Original: Instead, our approach is inherently interpretable [62], i.e., retrieving the patches first, and then using them to make classification decisions.
Revised: Instead, our approach is inherently interpretable [62], i.e., retrieving the patches first, and then using them to make classification decisions.

Original: While our binary classification task is adopted from [55], our study compares 4 different classifiers while Nguyen et al.
Revised: While our binary classification task is adopted from [55], our study compares 4 different classifiers while Nguyen et al.

Original: [55] instead tested a single classifier with multiple post-hoc explanations.
Revised: [55] instead tested a single classifier with multiple post-hoc explanations.

Original: Human studies Our study has 709 users in total, i.e. ∼ 60 users per method per dataset, which is substantially larger than that in most prior works.
Revised: Human studies Our study has 709 users in total, i.e. ∼ 60 users per method per dataset, which is substantially larger than that in most prior works.

Original: That is, ∼ 30 and 40 users per method participated in [55] and in [49], respectively while Adebayo et al.
Revised: That is, ∼ 30 and 40 users per method participated in [55] and in [49], respectively while Adebayo et al.

Original: [9] had 54 persons in total for the entire study of multiple methods.
Revised: [9] had 54 persons in total for the entire study of multiple methods.

Original: Human-AI teaming Human-AI collaboration is becoming more essential in the modern AI era [32].
Revised: Human-AI teaming Human-AI collaboration is becoming more essential in the modern AI era [32].

Original: A large body of prior works has investigated such collaboration in other domains (e.g., NLP [12, 76], healthcare [16] and others [33, 16, 78, 19]); however, only few works investigated human-AI collaboration in the image classification setting [55, 42, 24].
Revised: A large body of prior works has investigated such collaboration in other domains (e.g., NLP [12, 76], healthcare [16] and others [33, 16, 78, 19]); however, only few works investigated human-AI collaboration in the image classification setting [55, 42, 24].

Original: Some prior works predict when to defer the decision-making to humans [58, 37, 40].
Revised: Some prior works predict when to defer the decision-making to humans [58, 37, 40].

Original: However, by simply offloading some inputs to humans based on confidence scores, we achieve complementary human-AI team performance in both ImageNet and CUB.
Revised: However, by simply offloading some inputs to humans based on confidence scores, we achieve complementary human-AI team performance in both ImageNet and CUB.

Original: Previous works [12, 64] found that algorithmic explanations benefit human decision-making in general, but did not find XAI methods to yield team complementary performance [32], which we report in this work.
Revised: Previous works [12, 64] found that algorithmic explanations benefit human decision-making in general, but did not find XAI methods to yield team complementary performance [32], which we report in this work.

Original:  Discussion and Conclusion 
Revised:  Discussion and Conclusion Limitations

Original: Limitations Due to the limited amount of time and expensive cost of computation related to EMDCorr or CHM-Corr, we did not experiment on a wider range of OOD datasets (e.g., adversarial poses [10]).
Revised: Due to the limited amount of time and expensive cost of computation related to EMD-Corr or CHM-Corr, we did not experiment on a wider range of OOD datasets (e.g., adversarial poses [10]).

Original: We tested our methods on ImageNet-A [36], ObjectNet [13], and ImageNet-C [34] as well, but on a small scale of 5K-image sets (see Table. A2).
Revised: We tested our methods on ImageNet-A [36], ObjectNet [13], and ImageNet-C [34] as well, but on a small scale of 5K-image sets (see Table A2).

Original: As using online crowdworkers for the XAI human evaluation, we share the same limitations with [55, 9, 42].
Revised: As using online crowdworkers for the XAI human evaluation, we share the same limitations with [55, 9, 42].

Original: That is, despite our best efforts to minimize biases, the human data quality can be improved in highly-controlled laboratory conditions like in [27].
Revised: That is, despite our best efforts to minimize biases, the human data quality can be improved in highly-controlled laboratory conditions like in [27].

Original: Algorithm-wise, EMD-Corr and CHM-Corr are re-ranking methods and therefore run substantially slower than ResNet-50 (see Fig.
Revised: Algorithm-wise, EMD-Corr and CHM-Corr are re-ranking methods and therefore run substantially slower than ResNet-50 (see Fig.

Original: A10 for a speed comparison of all models).
Revised: A10 for a speed comparison of all models).

Original: Our work is the first attempt to: (1) study the effectiveness of patch-wise comparison in improving the robustness of deep image classifiers on ImageNet OOD benchmarks; (2) show the utility of visual correspondence-based explanations in helping users make more accurate image-classification decisions; (3) achieve human-AI complementary team performance in the image domain.
Revised: Our work is the first attempt to: (1) study the effectiveness of patch-wise comparison in improving the robustness of deep image classifiers on ImageNet OOD benchmarks; (2) show the utility of visual correspondence-based explanations in helping users make more accurate image-classification decisions; (3) achieve human-AI complementary team performance in the image domain.

Original: Acknowledgement
Revised: Acknowledgement

Original: The authors would like to thank Ken Stanley for the great idea of using correspondences as an explanation for image classification, leading to this work.
Revised: The authors would like to thank Ken Stanley for the great idea of using correspondences as an explanation for image classification, leading to this work.

Original: We also thank Thang Pham, Peijie Chen, and Hai Phan for feedback and discussions of the earlier results.
Revised: We also thank Thang Pham, Peijie Chen, and Hai Phan for feedback and discussions of the earlier results.

Original: AN was supported by the NSF Grant
Revised: AN was supported by the NSF Grant

Original: No. 1850117 & 2145767, and donations from NaphCare Foundation & Adobe Research.
Revised: No. 1850117 & 2145767, and donations from NaphCare Foundation & Adobe Research.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] .
Revised: [Yes] .

Original: Please see Sec.
Revised: Please see Sec.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] .
Revised: [Yes] .

Original: Please see Sec.
Revised: Please see Sec.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] .
Revised: [N/A] .

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] . 2.
Revised: [Yes] . 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] .
Revised: [Yes] .

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] .
Revised: [N/A] .

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] .
Revised: [No] .

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] .
Revised: [Yes] .

Original: Please see Sec.
Revised: Please see Sec.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] .
Revised: [Yes] .

Original: We reported whenever possible.
Revised: We reported whenever possible.

Original: See µ and σ in Table.
Revised: See µ and σ in Table 2 .

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No] . 4.
Revised: [No] . 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] .
Revised: [Yes] .

Original: We cited the authors and include the URLs .
Revised: We cited the authors and include the URLs .

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] .
Revised: [N/A] .

Original: We used publicly-available datasets and code .
Revised: We used publicly-available datasets and code .

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] .
Revised: [No] .

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] .
Revised: [Yes] .

Original: We explained to participants how their data will be used .
Revised: We explained to participants how their data will be used .

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] .
Revised: [Yes] .

Original: We checked and ensured that our data does not contain personally identifiable information or offensive content .
Revised: We checked and ensured that our data does not contain personally identifiable information or offensive content .

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Yes] .
Revised: [Yes] .

Original: See screenshots in Appendix E .
Revised: See screenshots in Appendix E .

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] .
Revised: [N/A] .

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Yes] .
Revised: [Yes] .

Original: See Sec.
Revised: See Sec.

Original: Our rate was $13.5/hr, higher than the Prolific recommended rate wage of $9.60/hr .Appendix for: Visual correspondence- based
Revised: Our rate was $13.5/hr, higher than the Prolific recommended rate wage of $9.60/hr .

Original: explanations improve AI robustness and human-AI team accuracy A Implementation details

Original: A.1 Fine-tuning iNaturalist-pretrained ResNet-50 for CUB

Original: To make a 200-way classifier using the ResNet-50 model from iNaturalist [4], we remove the 5089way classification head and add an average pooling layer followed by a linear feed-forward layer with 200 units.

Original: We keep all the initialization parameters unchanged and use the Adam optimizer [44] without any hyperparameter tuning.

Original: We train the new layer using the CUB training set forepochs.

Original: We do not train the intermediate layers since this backbone is shared among all methods (i.e., we freeze all the convolutional layers in the ResNet-50 model).

Original: The iNaturalist-pretrained ResNetmodel has slight difference compared to the PyTorch reference implementation [8].

Original: This network has 18 extra layers in the last convolutional blocks, but the spatial dimension matches the original

Original: ResNet-50 (i.e., 2048 × 7 × 7 ).

Original: A.2 Implementation details for kNN

Original: We implement a vanilla kNN classifier that operates at the deep feature space of ResNet-50.

Original: That is, given a query image Q , we sort all training-set images { G i } based on their distance D ( Q, G i ) , which is the cosine distance between the two corresponding image features f ( Q ) and f ( G i ) ∈

Original: Rat layer4 of ResNet-50, after avgpool (see code):

Original: [Equation]

Original: (1) where ⟨·⟩ is the dot product, and ∥·∥ is the L 2 norm operator.

Original: A

Original: .3 Implementation details for EMD-Corr

Original: Similar to [79, 29], we integrate the Earth Mover’s Distance (EMD) into a 2-stage hierarchical image retriever.

Original: In the first stage, we select the N images having the lowest cosine distance – { G i } with the query Q by the kNN classifier.

Original: Then, we sort these N images again (a.k.a re-ranking [29]) using patch-wise similarity derived from EMD.

Original: In the last step, the predicted label is determined by a majority vote of top- k images’ labels like the kNN classifier, where k ≤ N .

Original: In our classifier, we set k = 20 and N = 50 .

Original: Our patch-wise comparison algorithm in stage 2 (shown in Fig. 3a) is different from [29, 79, 75] as the similarity of an image pair is not determined by all possible patches.

Original: While the first stage retrieved images using global features, comparing only a few most similar patches by EMD offers benefits: (1) helping classifiers capture the distinctive image regions only (e.g., head-to-head comparison for birds); and (2) achieving human interpretability as looking at all possible pair-wise comparisons is impossible.

Original: We denote each patch-by-patch comparison as correspondence.

Original: The most similar patches between two images Q and G – both divided into M patches – are found as a set of 2-D coordinates L containing the highest values in a flow matrix F .

Original: Let Q =

Original: { ( q 1 , w q 1 ) , ( q 2 , w q 2 ) , · · · .

Original: ( q M , w q M ) } and G = { g 1 , w g 1 ) , ( g 2 , w g 2 ) , · · · .

Original: ( g M , w g M ) } denote two sets of non-overlapping image patches, g i and g j are the patch embeddings; and w q i and w g j are the corresponding importance assigned by a feature weighting algorithm (e.g., Cross Correlation used in [79]).

Original: We derive F = ( f ij ) ∈

Original: R M × M by minimizing the transport plan cost in Eq.

Original: [Equation]

Original: where f ij ≥ 0 and (cid:80)

Original: Mj =1 (cid:80)

Original: Mi

Original: =1 f ij = 1 .

Original: We use Eq.

Original: 1 to compute the ground distance d ij and run the Sinkhorn algorithm [21] for 100 iterations to seek the optimal transport plan F .

Original: To assign importance weights (i.e., w q i and w g j ), we use cross-correlation (CC) maps from [68].

Original: Finally, using F and D from Eq. 1, the EMD distance between Q and G is computed by Eq.

Original: Since we are interested in patch-wise comparison, the features used in stage 2 for Q and G are layer4 from [8].

Original: Our EMD-Corr classifier’s stage 2 solely relies on EMD distance for re-ranking instead of mixing

Original: EMD and cosine distance like in previous works [79, 29].

Original: [Equation]

Original: A.4 Implementation details for CHM-Corr classifier

Original: Similar to the EMD-Corr classifier, this classifier also consists of 2 stages – selecting N most similar images to the query Q by the kNN classifier, followed by a correspondence-based re-ranking algorithm.

Original: For re-ranking, we propose to use a Convolutional Hough Matching network (CHM) [51] to first infer semantic correspondences between Q and G , then calculate the similarity score between these two images based on a subset of correspondences.

Original: The re-ranking algorithm starts with dividing both Q and G into M patches, resulting in two set of Q = { q 1 , q 2 , · · · .q

Original: M } and G = { g 1 , g 2 , · · · .g M } image patches.

Original: To find the semantic correspondences between two images, we make use of the CHM network to transfer keypoints from the query image Q to image G .

Original: The CHM network finds correspondence between two given images in three stages: Feature extraction and correlation computation, Hough matching, and keypoint transfer.

Original: In the first stage, the CHM network extracts features from multiple layers of a ResNet-101 network to construct a set of multiscale features { ( F Q , F G ) }

Original: Ss =1 .

Original: The feature volume is then used to construct a correlation tensor by comparing all possible pairs in the feature space of two images.

Original: In the second stage, the correlation tensor is fed into a Convlutioanl Hough Matching (CHM) layer to perform Hough voting in the space of translation and scale to find candidate matches between two images.

Original: In the last stage, a kernel soft-argmax [46] is applied to the output of the CHM layer to create a dense flow field, and then correspondence keypoints are extracted using a soft sampler.

Original: After finding visual correspondence between two images, we assign an importance weight w i,j for the pair ( q i , g j ) using cross-correlation maps from [68].

Original: Finally, the distance between Q and G is the average distance between 5 patch pairs with the lowest cosine distance.

Original: We use the reference implementation of the Convolutional Hough Matching Network pretrained on the

Original: PF-Pascal Dataset [30].

Original: There are three variations of CHM networks depending on parameters sharing strategy, i.e., psi , iso , and full .

Original: Our ablation study (Appendix B.3) shows similar performance on a 5K subset of the ImageNet dataset.

Original: We select psi with a threshold of T = 0 .

Original: 55 for the CHM-Corr classifier.

Original: The CHM network requires a set of initial keypoints on the source image, i.e., a set of keypoint on the query image Q .

Original: Although some datasets come with this annotation information, generally, this information is not available.

Original: To have a comparable classifier with our EMD-Corr classifier, we discretize an image into a 7 × 7 grid, resulting in 49 non-overlapping patches.

Original: For each patch, we pick a point at its center.

Original: For assigning importance weight w i,j to ( q i , g j ) pair, we first calculate the cross-correlation map between the two images Q and G .

Original: Calculating a cross-correlation map using the last convolutional layer of the ResNet-50 model will result in two 7 × 7 maps for each Q and G .

Original: For assigning importance weights, we binarize the cross-correlation for Q , using a threshold of T = 0 .

Original: 55 , i.e., we zero out all pairs in the non-salient part according to Q , by setting their importance weights to 0 , and for the remaining patches, we set the weights to 1 .

Original: After removing non-salient patch pairs in the last stage, we calculate the cosine similarity between pair ( q i , g j ) using the corresponding feature volume in the last convolutional layer of the ResNetmodel.

Original: The similarity score is the average similarity between top 5 pairs with the highest cosine similarly.

Original: A.5 Generating Adversarial Patch dataset

Original: Brown et al.

Original: [15] generated a universal adversarial patch to fool image classifiers to recognize everything as toaster .

Original: This patch misleads models’ attention, overly looking at the most salient item while ignoring the remaining pixels.

Original: We apply this attack on ImageNet validation set, resulting in 50K Adversarial Patch images of 240 × 240 px.

Original: The patches are circles with the size of 5 % the input image, targeting on ResNet-50 [31] classifying everything as toaster with target confidence of 90%.

Original: The maximum attack iteration for each sample is 500.

Original: We only train to optimize the adversarial patch on ImageNet validation set for one epoch and save the immediate samples for the dataset.

Original: We adapt the code from [2] and make minor modifications.

Original: To obtain our Adversarial Patch dataset, from the main repository, you can run the below command to generate the dataset or download the dataset here.

Original: cd datasets/adversarial-patch/ python make_patch.py --cuda --epochs 1 --patch_size 0.05 --max_count--netClassifier resnet50 --patch_type circle --train_size-

Original: [Table]

Original: B Ablation study and small-scale experiments on ImageNet OODs B.1 Different hyperparameters for EMD

Original: [Table]

Original: B.2 Performance of classifiers on a 5K subset of different datasets

Original: Table A2 contains details about the performance of different classifiers on a 5K subset of various

Original: OOD datasets.

Original: [Table]

Original: [Table]

Original: ImageNet-R [35]

Original: [Table]

Original: ObjectNet [13]

Original: [Table]

Original: ImageNet Sketch [72]

Original: [Table]

Original: ImageNet-A [36]

Original: [Table]

Original: [Table]

Original: DAmageNet [18]

Original: [Table]

Original: ImageNet-C Gaussian noise (Level 1) [34]

Original: [Table]

Original: ImageNet-C Gaussian blur (Level 1) [34]

Original: [Table]

Original: 63] with different CHM

Original: [Table]

Original: [Table]

Original: C Runtime comparison between all methods

Original: In this section, we provide a runtime analysis of all classifiers on a batch of 1000 random queries.

Original: For each classifier, we run the classification five times and report the average and standard deviation.

Original: We use a single NVIDIA V100 GPU with 16 gigabytes of memory to perform our benchmarks.

Original: Here we also provide a FAISS [39] implementation of the kNN classifier, which is significantly faster than the naive GPU implementation for the nearest neighbor search problem.

Original: The FAISS version of kNN requires a one-time preprocessing to extract embeddings of the training set.

Original: This process takes just a few minutes for the CUB dataset, which has only 5.9K images.

Original: For ImageNet, which consists of 1.2 million images, we use a single NVIDIA A100 (40 GB) to extract and cache the embeddings on disk.

Original: This process takes less than 90 minutes, and the resulting cache file takes 9.

Original: GB of disk space.

Original: We also use Linux time tool to calculate the total memory usage of kNN using

Original: FAISS during the inference.

Original: The peak memory performance ( Maximum resident set size ) for the 1000 images is around 31 GB.

Original: [Table]

Original: D Sample explanations

Original: This section contains sample visualization for kNN, EMD-Corr, and CHM-Corr classifiers.

Original: D.1 kNN Explanations

Original: [Figure]

Original: D.2 EMD-NN

Original: [Figure]

Original: D.3 EMD-Corr Explanations Query

Original: Explanations Query

Original: D.5 CHM-Corr

Original: Explanations

Original: Query

Original: E Sample screens and training examples for human studies E.1 Sample screens from human studies

Original: [Figure]

Original: E.2 Sample groundtruth cases used in the Validation phase of our CUB human studies

Original: Below are example cases that we manually choose to be “groundtruth” in order to control for user quality during the validation phase.

Original: [Figure]

Original: (a) ResNet-50—no explanations provided.

Original: (b) kNN nearest-neighbor explanation.

Original: (c) EMD-Corr explanation.

Original: [Figure]

Original: (a) EMD-NN explanation.

Original: (b) EMD-Corr explanation.

Original: F Human-AI team performance analysis

Original: This section provides more details about AI and team performance.

Original: F.1 Defining the difficulty level of queries

Original: To further understand the performance of the classifiers in our study, we categorize each query image into Easy, Medium

Original: [Table]

Original: and Hard categories based on the model’s confidence score and the correctness of the top-1 label (see Table A5).

Original: This breakdown allows us to analyze model and user behaviors in a specific level of difficulty

Original: [Table]

Original: [Table]

Original: F

Original: The acceptance and rejection ratios

Original: In Table A6 we provide details about whether users accepted or rejected AI’s decisions for each type of classifier.

Original: [Table]

Original: Table A7 shows the ratio of accept and reject based on the difficulty level described in Sec. F.1.

Original: [Table]

Original: F.3 Time performance of users

Original: Fig.

Original: A10 shows the average time distribution to finish each trail per method.

Original: [Figure]

Original: Figure A11 shows the breakdown of user accuracy based on the correctness of AI predictions on

Original: ImageNet-ReaL and CUB datasets.

Original: [Figure]

Original: F.5 Human performance analysis based on the difficulty of the query

Original: In this section, we calculate the average user accuracy based on the difficulty level of the query (Described in Sec F.1) and the correctness of AI’s prediction

Original: [Figure]

Original: g

Original: [Figure]

Original: hu m

Original: [Figure]

Original: n a

Original: [Figure]

Original: c y

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: N u m b e r o f Q u e r

Original: i e s

Original: [Figure]

Original: N u m b e r o f Q u e r

Original: i e s

Original: [Figure]

Original: F.6 Analysis of Hard images for humans in the ImageNet task

Original: This section provides the analysis to understand what kind of queries are hard for humans, i.e., for what types of images users can not correctly accept or reject the AI’s decision.

Original: To this end, we filter the queries with a mean user’s accuracy of 0 .

Original: 25 or below.

Original: Figure A14 shows the distribution of Hard images for humans based on the classifier and the classifier’s correctness.

Original: [Figure]

Original: To better understand what types of images are more challenging for humans, we automatically create supergroups for ImageNet class members.

Original: All 1,000 classes of ImageNet are a subgroup of entity - n00001740 class in the WordNet glossary [50].

Original: To create groups with uniform sizes, we start from the entity root node and break up each class to its sub-classes recursively; in each iteration, we pick the supergroup with the largest number of classes.

Original: Here we report the parent class of queries afteriterations and for queries with only one ImageNet ReaL label.

Original: Using this automated procedure, we can see that the majority of hard images for humans fall into the carnivore category, which is a supergroup for cats, lions, dogs, wolves, etc.

Original: Details about each parent class and its ImageNet class members can be found in Table A8.

Original: c a r n i v o r e un g u l a t e c

Original: o m m o d

Original: i t y b i r d r e p t il e i n v e r t

Original: e b r a t e c o nn e c t i o n m a tt e r c o v e r i n g s t r u c t u r e i m p l e m e n t c o n v e y a n c e l a g o m o r p h p

Original: a c h y d

Original: e r m d e v

Original: i c e t o il e t r y p

Original: r

Original: i m a t e n a t u r a l _ o b

Original: j e c t d e c o r a t i o n a q u a t i c _ v e r t e b r a t e c o n t a i n e r f un g u s a b s t r a c t

Original: i o n f a b r i c a m p h

Original: i b i a n f u r n i s h

Original: i n g e q u

Original: i p m e n t g e

Original: o l o g i c a l _ f o r m a t

Original: i o n s y s t e m r o d e n t p l a y t

Original: h

Original: i n g

Original: [Figure]

Original: N u m b e r o f Q u e r

Original: i e s

Original: [Figure]

Original: [Table]

Original: F.7 Analysis of Hard images for humans in the bird classification task

Original: Similar to the analysis we conducted for ImageNet in Sec.

Original: F.6, here we analyze the confusing bird types for humans.

Original: Here we filter queries with mean user accuracy below 0 .

Original: Figure A16 shows the distribution of the most challenging samples for humans based on different classifiers’ correctness.

Original: [Figure]

Original: Table A9 shows the top 5 confusing bird types for human users.

Original: Each row in this table shows how many users failed to reject AI’s prediction when providing different kinds of explanations.

Original: [Table]

Original: F.8 Accepting AI’s wrong decision

Original: This section shows samples for which users incorrectly accepted the incorrect AI prediction.

Original: F.8.1 Accepting the wrong kNN Classifier’s prediction

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: F.9 When explanations fool users

Original: This section provides clear evidence that explanations have the potentials to fool human users.

Original: Both

Original: ResNet-50 and EMD-Corr misclassified an image of tow truck into cab .

Original: When asking a user to accept or reject this particular misclassification, they act differently based on the provided explanation.

Original: A total of 6 users who saw the query without any visual explanation were able to correctly reject

Original: AI’s decision, while 3 out of 6 (50%) users who received either EMD-NN or EMD-Corr explanation incorrectly accepted the decision.

Original: [Figure]

Original: G Classification accuracy of Human-AI teams

Original: This section contains details on the Human and AI team’s complementary performance on different confidence thresholds T .

Original: For each threshold T

Original: [Table]

Original: we divide the set of 44K ImageNet-ReaL validation images into two groups; (1) images where the AI’s confidence is greater or equal to > = T and (2) images where the AI’s confidence is less

Original: than < T .

Original: In the first group, we only consider AI’s decision.

Original: For the second group, we ask a human user to judge AI’s predicted label,

Original: i

Original: [Table]

Original: e., a human user should accept or reject AI’s classification.

Original: [Table]

Original: Aggregating Human and AI (%)

Original: [Table]

Original: n/a 0.

Original: n/a n/a 0.

Original: n/a

Original: [Table]

Original: [Table]

Original: n/a n/a 0.

Original: [Table]

Original: [Table]

Original: [Table]

Original: n/a n/a 83.

Original: n/a n/a 0.

Original: [Table]

Original: H Analysis for CUB H.1 Correspondences help users to reject wrong AI prediction

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: H.2 Diversity of images in kNN and, EMD-Corr, and CHM-Corr explanations

Original: We hypothesize that when the AI prediction is wrong, the diversity among the five nearest neighbors of kNN differs from EMD-Corr and CHM-Corr, leading to users rejecting the decision.

Original: To this end, we calculated LPIPS and MS-SSIM metrics between all possible pairs of explanations on the relevant queries.

Original: [Figure]

Original: H.3 When the user rejects the correct AI prediction

Original: This section provides a brief qualitative explanation for the cases where users incorrectly rejected a correct AI prediction.

Original: [Figure]

Original: I Comparing explanation methods

Original: This section compares explanations provided by kNN, EMD-Corr

Original: [Figure]

Original: and CHM-Corr for various sets of queries.

Original: I.1 ImageNet samples

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: J Controlling keypoints in CHM-Corr+ for the CUB dataset

Original: Here, we compare CHM-Corr and CHM-Corr+ classifiers to understand the low performance of

Original: CHM-Corr+.

Original: Using a set of five keypoints may not help CHM-Corr+ focus on the right patches.

Original: Sometimes, the five provided keypoints are not among the discriminative features to correctly classify a bird.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: L Removing duplicated images from the ImageNet validation set

Original: Some of the images from the ImageNet validation set are also present in the training set.

Original: For the human study, we excluded such images from our study.

Original: Figure A45, shows some of these samples along with their five nearest neighbor images from the training set.

Original: [Figure]

Original: ILSVRC

Original: [Figure]

Original: [Figure]

Original: JPEG

Original: [Figure]

Original: ILSVRC2012_val_

Original: [Figure]

Original: JPEG

Original: [Figure]

Original: ILSVRC2012_val_

Original: [Figure]

Original: JPEG

Original: [Figure]

Original: M Human-AI teams outperform both AIs alone and humans alone

Original: In Sec. 3.5, we find that user classification accuracy can be improved when humans are provided with AI predictions and explanations.

Original: Here, we leverage the same data collected from the previous

Original: ImageNet-ReaL and CUB human studies (Secs.

Original: 3.4 and 3.5) to estimate the accuracy of a human-AI team that allows both humans and AIs to make final decisions (Fig. 4; Model 2).

Original: That is, AIs make binary

Original: Yes/No predictions on all the X % of query images that they assign a high confidence score ≥ T where T ∈ [0 , 1] (Fig. 4b).

Original: Given these images and AI predictions, we compute an accuracy score acc AI .

Original: For the set of remaining images (i.e. whose AI confidence is < T ), we take their user predictions and also compute an accuracy score acc human .

Original: We define the human-AI team accuracy acc team as:

Original: [Equation]

Original: As the interaction model 2 is more practical and scalable, it is interesting to test how the acc team compares with the accuracy when users or AIs alone classify all images (i.e. when X = 0 or 100 ).

Original: Experiment For each classifier (ResNet-50, kNN, EMD-Corr, and CHM-Corr), we use a 2K-image held-out subset of the ImageNet-ReaL validation 3 set to find an optimal threshold T that maximizes the classifier’s binary classification accuracy.

Original: Then, we use the remaining ∼ 42K ImageNet-ReaL validation images for testing.

Original: For CUB, we tune T using 1K test images and test on the remaining ∼ 4.7K test-set images.

Original: For both ImageNet and CUB, we did not use the training sets to tune T because the top-1 neighbors retrieved by kNN would be identical to the query all the time, biasing the AIs as well as humans when they perform classification.

Original: After obtaining an AI accuracy score for each value of T ∈ { 0 . 05 , 0 . 10 , 0 .

Original: 95 } , we find the best acc team (at an optimal T ∗ ) and repeat the same process to find the best AI-only accuracy.

Original: More details of the sweeps are in Appendix G.

Original: Results First, across all four classifiers and two datasets, AI-only accuracy is consistently higher than human-only accuracy (Table 3 vs. Table 2).

Original: That is, letting users make all the AI-assisted decisions is both more labor-intensive and less accurate compared to letting AIs classify all the data themselves.

Original: This result is consistent with the prior studies that find AIs to outperform humans [28, 78, 59, 26] (see [12] for a summary).

Original: Second, interestingly, human-AI teams consistently outperform the AIs alone (Table 3) and humans alone (Table 2).

Original: That is, lay-users may be considered “expert” on ImageNet’s everyday objects and therefore, when teaming up with humans to form human-AI teams, the accuracy substantially increases on average by +3.88 (Table 3).

Original: On CUB, which is more challenging to lay-users, this benefit of teaming up with users is smaller (Table 3; +0.96)

Original: Third, among four classifiers, ResNet-50 yields the highest human-AI team accuracy on both

Original: ImageNet-ReaL and CUB (Table 3).

Original: However, the variance in team accuracy across the classifiers is small.

Original: Our results interestingly imply that while there is a significant evidence that Corr explanations are useful to the AI-assisted decision making of humans in the interaction model(Table 2; CUB), such benefits of XAI models average out in the interaction model 2.