Original: Harmonizing the object recognition strategies of deep neural networks with humans
Revised: Harmonizing the object recognition strategies of deep neural networks with humans

Original: Abstract
Revised: Abstract

Original: The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence.
Revised: The many successes of deep neural networks (DNNs) over the past decade have largely been driven by computational scale rather than insights from biological intelligence.

Original: Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition.
Revised: Here, we explore if these trends have also carried concomitant improvements in explaining the visual strategies humans rely on for object recognition.

Original: We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects.
Revised: We do this by comparing two related but distinct properties of visual strategies in humans and DNNs: where they believe important visual features are in images and how they use those features to categorize objects.

Original: Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition.
Revised: Across 84 different DNNs trained on ImageNet and three independent datasets measuring the where and the how of human visual strategies for object recognition on those images, we find a systematic trade-off between DNN categorization accuracy and alignment with human visual strategies for object recognition.

Original: State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves .
Revised: State-of-the-art DNNs are progressively becoming less aligned with humans as their accuracy improves .

Original: We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns
Revised: We rectify this growing issue with our neural harmonizer: a general-purpose training routine that both aligns

Original: DNN and human visual strategies and improves categorization accuracy.
Revised: DNN and human visual strategies and improves categorization accuracy.

Original: Our work represents the first demonstration that the scaling laws [1–3] that are guiding the design of DNNs today have also produced worse models of human vision.
Revised: Our work represents the first demonstration that the scaling laws [1–3] that are guiding the design of DNNs today have also produced worse models of human vision.

Original: We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.
Revised: We release our code and data at https://serre-lab.github.io/Harmonization to help the field build more human-like DNNs.

Original:  Introduction
Revised:  Introduction

Original: Rich Sutton stated [4] that the bitter lesson “from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
Revised: Rich Sutton stated [4] that the bitter lesson “from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”

Original: Deep learning has been the standard approach to object categorization problems ever since the paradigm shifting success of AlexNet [5] on the ImageNet [6] benchmark a decade ago.
Revised: Deep learning has been the standard approach to object categorization problems ever since the paradigm shifting success of AlexNet [5] on the ImageNet [6] benchmark a decade ago.

Original: As deep neural network (DNN) performance has continued to improve in the intervening years, Sutton’s lesson has become more fitting than ever with recent networks rivaling and likely outperforming humans on the benchmark [7] through brute-force computational scale: increasing the number of network parameters and number of images used for training orders-of-magnitude beyond AlexNet [1–3].
Revised: As deep neural network (DNN) performance has continued to improve in the intervening years, Sutton’s lesson has become more fitting than ever with recent networks rivaling and likely outperforming humans on the benchmark [7] through brute-force computational scale: increasing the number of network parameters and number of images used for training orders-of-magnitude beyond AlexNet [1–3].

Original: While the successes of so-called “scaling laws” are undeniable, this singular focus on performance in the field has side-stepped an equally important question that will govern the utility of object recognition models for the brain sciences and industry applications alike: are the visual strategies learned by DNNs aligned with those used by humans?
Revised: While the successes of so-called “scaling laws” are undeniable, this singular focus on performance in the field has side-stepped an equally important question that will govern the utility of object recognition models for the brain sciences and industry applications alike: are the visual strategies learned by DNNs aligned with those used by humans?

Original: The visual strategies that mediate object recognition in humans can be decomposed into two related but distinct processes: identifying where the important features for object recognition are in a scene, and determining how to integrate the selected features into a categorical decision [8,9].
Revised: The visual strategies that mediate object recognition in humans can be decomposed into two related but distinct processes: identifying where the important features for object recognition are in a scene, and determining how to integrate the selected features into a categorical decision [8,9].

Original: It has been known for nearly a century [10–13] that different humans attend to similar locations when asked to find and recognize objects.
Revised: It has been known for nearly a century [10–13] that different humans attend to similar locations when asked to find and recognize objects.

Original: After selecting these important features, human observers are also consistent in how they use those features to categorize objects – the inclusion of a few pixels in an image can be the difference between recognizing an object or not [9,14].
Revised: After selecting these important features, human observers are also consistent in how they use those features to categorize objects – the inclusion of a few pixels in an image can be the difference between recognizing an object or not [9,14].

Original: Has the past decade of DNN development produced any models that are aligned with these human visual strategies for object recognition?
Revised: Has the past decade of DNN development produced any models that are aligned with these human visual strategies for object recognition?

Original: Such a model could transform cognitive science by supporting a better mechanistic understanding of how vision works.
Revised: Such a model could transform cognitive science by supporting a better mechanistic understanding of how vision works.

Original: More human-like models of object recognition would also resolve the problems with predictablity and interpretablity of DNNs [15–18], and control their alarming tendency to rely on “shortcuts” and dataset biases to perform well on tasks [19].
Revised: More human-like models of object recognition would also resolve the problems with predictablity and interpretablity of DNNs [15–18], and control their alarming tendency to rely on “shortcuts” and dataset biases to perform well on tasks [19].

Original: In this work, we perform the first large-scale and systematic comparison of the visual strategies of DNNs and humans for object recognition on ImageNet.
Revised: In this work, we perform the first large-scale and systematic comparison of the visual strategies of DNNs and humans for object recognition on ImageNet.

Original: Contributions.
Revised: Contributions.

Original: In order to compare human and DNN visual strategies, we first turn to the human feature importance maps collected by Linsley et al.
Revised: In order to compare human and DNN visual strategies, we first turn to the human feature importance maps collected by Linsley et al.

Original: Their datasets, ClickMe and Clicktionary , contain maps of nearly 200,000 unique images in ImageNet that highlight the visual features humans believe are important for recognizing them.
Revised: Their datasets, ClickMe and Clicktionary , contain maps of nearly 200,000 unique images in ImageNet that highlight the visual features humans believe are important for recognizing them.

Original: These datasets amount to a reverse inference on where important visual features are in ImageNet images (Fig. 1).
Revised: These datasets amount to a reverse inference on where important visual features are in ImageNet images (Fig. 1).

Original: We complement these datasets with new psychophysics experiments that directly test how important visual features are used for object recognition (Fig. 1).
Revised: We complement these datasets with new psychophysics experiments that directly test how important visual features are used for object recognition (Fig. 1).

Original: As DNN performance has increased on ImageNet, their alignment with human visual strategies captured in these datasets has worsened.
Revised: As DNN performance has increased on ImageNet, their alignment with human visual strategies captured in these datasets has worsened.

Original: This trade-off is found overdifferent DNNs representing all popular model classes – from those trained for adversarial robustness to those pushing the scaling laws in network capacity and training data.
Revised: This trade-off is found overdifferent DNNs representing all popular model classes – from those trained for adversarial robustness to those pushing the scaling laws in network capacity and training data.

Original: To summarize our findings:
Revised: To summarize our findings:

Original: • The trade-off between DNN object recognition accuracy and alignment with human visual strategies replicates across three unique datasets: ClickMe [20], Clicktionary [21]
Revised: • The trade-off between DNN object recognition accuracy and alignment with human visual strategies replicates across three unique datasets: ClickMe [20], Clicktionary [21]

Original: and our psychophysics experiments.
Revised: and our psychophysics experiments.

Original: • We shift this trade-off with our neural harmonizer, a novel drop-in module for co-training any DNN to align with human visual strategies while also achieving high task accuracy.
Revised: • We shift this trade-off with our neural harmonizer, a novel drop-in module for co-training any DNN to align with human visual strategies while also achieving high task accuracy.

Original: Harmonized DNNs learn visual strategies that are significantly more aligned with humans than any other DNN we tested .
Revised: Harmonized DNNs learn visual strategies that are significantly more aligned with humans than any other DNN we tested .

Original: • We release our data and code at https://serre-lab.github.io/Harmonization/ to help the field tackle the growing misalignment between DNNs and humans.
Revised: • We release our data and code at https://serre-lab.github.io/Harmonization/ to help the field tackle the growing misalignment between DNNs and humans.

Original:  Related work
Revised:  Related work

Original: Where are the important features?
Revised: Where are the important features?

Original: How are the discovered features used?
Revised: How are the discovered features used?

Original: [Figure]
Revised: [Figure]

Original: Do DNNs explain human visual perception?
Revised: Do DNNs explain human visual perception?

Original: Despite the continued success of DNNs on computer vision benchmarks, there are conflicting accounts on their ability to explain human vision.
Revised: Despite the continued success of DNNs on computer vision benchmarks, there are conflicting accounts on their ability to explain human vision.

Original: On the one hand, there is evidence that DNNs are improving as models of human visual perception on challenging tasks, such as recognizing objects obscured by noise [23].
Revised: On the one hand, there is evidence that DNNs are improving as models of human visual perception on challenging tasks, such as recognizing objects obscured by noise [23].

Original: On the other hand, there is also evidence that DNNs struggle to explain perceptual phenomena in human vision like contextual illusions [24], perceptual grouping [19,25,26], and categorical prototypes [27].
Revised: On the other hand, there is also evidence that DNNs struggle to explain perceptual phenomena in human vision like contextual illusions [24], perceptual grouping [19,25,26], and categorical prototypes [27].

Original: Others have found differences between human attention data and DNN models of visual attention [20,28].
Revised: Others have found differences between human attention data and DNN models of visual attention [20,28].

Original: Moreover,
Revised: Moreover,

Original: DNNs have stopped improving as models of the ventral visual system in humans and primates over recent years.
Revised: DNNs have stopped improving as models of the ventral visual system in humans and primates over recent years.

Original: While the original theory was that model explanations of object-evoked neural activity patterns improved alongside model categorization accuracy [29], recent large-scale DNNs are worse at explaining neural data than older ones with lower ImageNet accuracy [30]
Revised: While the original theory was that model explanations of object-evoked neural activity patterns improved alongside model categorization accuracy [29], recent large-scale DNNs are worse at explaining neural data than older ones with lower ImageNet accuracy [30]

Original: What are the visual strategies underlying human object recognition?
Revised: What are the visual strategies underlying human object recognition?

Original: Ever since its inception, a goal of vision science has been to characterize the neural processes supporting object recognition in humans.
Revised: Ever since its inception, a goal of vision science has been to characterize the neural processes supporting object recognition in humans.

Original: It has been discovered that object recognition can be decomposed into different processing stages that emerge over time [8,31–36], where the earliest stage is associated with processing through feedforward connections in the visual system, and the later stage is associated with processing through feedback connections.
Revised: It has been discovered that object recognition can be decomposed into different processing stages that emerge over time [8,31–36], where the earliest stage is associated with processing through feedforward connections in the visual system, and the later stage is associated with processing through feedback connections.

Original: Since the DNNs used today mostly rely on feedforward connections, it is likely that they are better models for that rapid feedforward phase of processing than the subsequent feedback phase [33,37].
Revised: Since the DNNs used today mostly rely on feedforward connections, it is likely that they are better models for that rapid feedforward phase of processing than the subsequent feedback phase [33,37].

Original: To maximize the likelihood that the visual strategies learned by DNNs align with those used by humans, our experiments focus on the visual strategies of rapid feedforward object recognition in humans.
Revised: To maximize the likelihood that the visual strategies learned by DNNs align with those used by humans, our experiments focus on the visual strategies of rapid feedforward object recognition in humans.

Original: Most closely related to our work, are studies of “top-down” image saliency and where category diagnostic visual features are in images.
Revised: Most closely related to our work, are studies of “top-down” image saliency and where category diagnostic visual features are in images.

Original: These studies typically involve asking participants to search for an object in an image, or find visual features that are diagnostic for an object’s category or identity [10–13, 20, 22, 38].
Revised: These studies typically involve asking participants to search for an object in an image, or find visual features that are diagnostic for an object’s category or identity [10–13, 20, 22, 38].

Original: In our work, we complement these descriptions of where important features are in images with psychophysics testing how those features are used to categorize objects.
Revised: In our work, we complement these descriptions of where important features are in images with psychophysics testing how those features are used to categorize objects.

Original: Comparing visual strategies of humans and machines.
Revised: Comparing visual strategies of humans and machines.

Original: As methods in explainable artificial intelligence have developed over the past decade, they have opened up opportunities for comparing the visual regions selected by humans and DNNs when solving tasks.
Revised: As methods in explainable artificial intelligence have developed over the past decade, they have opened up opportunities for comparing the visual regions selected by humans and DNNs when solving tasks.

Original: Many of these comparisons have focused on human image saliency measurements captured by eye tracking or mouse clicks during passive or active viewing [20,22,39–42].
Revised: Many of these comparisons have focused on human image saliency measurements captured by eye tracking or mouse clicks during passive or active viewing [20,22,39–42].

Original: Others have compared categorical representation distances [40, 43] or combined those distances with measures of human attention [28].
Revised: Others have compared categorical representation distances [40, 43] or combined those distances with measures of human attention [28].

Original: The most direct comparisons between human and DNN visual strategies involved analyzing the minimal image patches needed to recognize objects [9,44,45].
Revised: The most direct comparisons between human and DNN visual strategies involved analyzing the minimal image patches needed to recognize objects [9,44,45].

Original: However, these studies were limited and compared humans with older DNNs on tens of images.
Revised: However, these studies were limited and compared humans with older DNNs on tens of images.

Original: To the best of our knowledge, the largest-scale evaluation of human and DNN visual strategies relied on the ClickMe dataset to compare visual regions preferred by humans and attention models trained for object recognition [20].
Revised: To the best of our knowledge, the largest-scale evaluation of human and DNN visual strategies relied on the ClickMe dataset to compare visual regions preferred by humans and attention models trained for object recognition [20].

Original: What is noticeably missing from each of these studies is an large-scale analysis spanning many images and models of how human and
Revised: What is noticeably missing from each of these studies is an large-scale analysis spanning many images and models of how human and

Original: DNN alignment has changed as a function of model performance.
Revised: DNN alignment has changed as a function of model performance.

Original: Improving the correspondence between humans and machines.
Revised: Improving the correspondence between humans and machines.

Original: Inconsistencies between human and DNN representations can be resolved by directly training models to act more like humans.
Revised: Inconsistencies between human and DNN representations can be resolved by directly training models to act more like humans.

Original: DNNs have been trained to have more human-like attention, or human-like representational distances in their output layers [20,40,43,46,47].
Revised: DNNs have been trained to have more human-like attention, or human-like representational distances in their output layers [20,40,43,46,47].

Original: Here, we add to these successes with the neural harmonizer, a training routine that automatically aligns the visual strategies (Fig. 1) of any two observers by minimizing the dissimilarity of their decision explanations.
Revised: Here, we add to these successes with the neural harmonizer, a training routine that automatically aligns the visual strategies (Fig. 1) of any two observers by minimizing the dissimilarity of their decision explanations.

Original:  Methods
Revised:  Methods

Original: Human feature importance datasets.
Revised: Human feature importance datasets.

Original: We focused on the ImageNet dataset to compare the visual strategies of humans and DNNs for object recognition at scale.
Revised: We focused on the ImageNet dataset to compare the visual strategies of humans and DNNs for object recognition at scale.

Original: We relied on the two significant efforts for gathering feature importance data from humans on ImageNet: the Clicktionary [22] and
Revised: We relied on the two significant efforts for gathering feature importance data from humans on ImageNet: the Clicktionary [22] and

Original: ClickMe [20] games, which use slightly different methods to collect their data.
Revised: ClickMe [20] games, which use slightly different methods to collect their data.

Original: Both games begin with the same basic setup: two players work together to locate features in an object image that they believe are important for categorizing it.
Revised: Both games begin with the same basic setup: two players work together to locate features in an object image that they believe are important for categorizing it.

Original: As one of the players selects important image regions, those regions are filled into a blank canvas for the other observer to see and categorize the image as quickly as possible.
Revised: As one of the players selects important image regions, those regions are filled into a blank canvas for the other observer to see and categorize the image as quickly as possible.

Original: In Clicktionary [22], both players are humans, whereas in ClickMe [20], the player selecting features is a human and the player recognizing images is a DNN (VGG16 [48]).
Revised: In Clicktionary [22], both players are humans, whereas in ClickMe [20], the player selecting features is a human and the player recognizing images is a DNN (VGG16 [48]).

Original: For both games, feature importance maps depicting the average object category diagnosticity of every pixel was computed as the probability of it being clicked by a participant.
Revised: For both games, feature importance maps depicting the average object category diagnosticity of every pixel was computed as the probability of it being clicked by a participant.

Original: In total, Clicktionary [22] contained feature importance maps for 200 images from the ImageNet validation set, whereas ClickMe [20] contained feature importance maps for a non-overlapping set of 196,499 images from ImageNet training and validation sets.
Revised: In total, Clicktionary [22] contained feature importance maps for 200 images from the ImageNet validation set, whereas ClickMe [20] contained feature importance maps for a non-overlapping set of 196,499 images from ImageNet training and validation sets.

Original: Thus, ClickMe has far more data than Clicktionary , but Clicktionary data has more reliable human feature importance data than ClickMe .
Revised: Thus, ClickMe has far more data than Clicktionary , but Clicktionary data has more reliable human feature importance data than ClickMe .

Original: Our experiments measure the alignment between human and DNN visual strategies using ClickMe and Clicktionary feature importance maps captured on the ImageNet validation set.
Revised: Our experiments measure the alignment between human and DNN visual strategies using ClickMe and Clicktionary feature importance maps captured on the ImageNet validation set.

Original: As we describe in §4, ClickMe feature importance maps from the ImageNet training set are used to implement our neural harmonizer.
Revised: As we describe in §4, ClickMe feature importance maps from the ImageNet training set are used to implement our neural harmonizer.

Original: Psychophysics participants and dataset.
Revised: Psychophysics participants and dataset.

Original: We complemented the feature importance maps from
Revised: We complemented the feature importance maps from

Original: Clicktionary and ClickMe with psychophysics experiments on rapid visual categorization.
Revised: Clicktionary and ClickMe with psychophysics experiments on rapid visual categorization.

Original: We recruited 199 participants from Amazon Mechanical Turk ( mturk.com ) to complete the experiments.
Revised: We recruited 199 participants from Amazon Mechanical Turk ( mturk.com ) to complete the experiments.

Original: Participants viewed a psychophysics dataset consisting of the 100 animal and 100 non-animal images in the Clicktionary game taken from the ImageNet validation set [22].
Revised: Participants viewed a psychophysics dataset consisting of the 100 animal and 100 non-animal images in the Clicktionary game taken from the ImageNet validation set [22].

Original: We used the feature importance maps for each image as masks for the object images, allowing us to control the proportion of important features observers were shown when asked to recognize objects (Fig. 5a).
Revised: We used the feature importance maps for each image as masks for the object images, allowing us to control the proportion of important features observers were shown when asked to recognize objects (Fig. 5a).

Original: We generated versions of each image that reveal anywhere between 1% to 100% (at log-scale spaced intervals) of the important object pixels against a phase scrambled noise background (see Appendix §1 for details on mask generation).
Revised: We generated versions of each image that reveal anywhere between 1% to 100% (at log-scale spaced intervals) of the important object pixels against a phase scrambled noise background (see Appendix §1 for details on mask generation).

Original: The total number of revealed pixels was equal for every image at a given level of image masking, and the revealed pixels were centered against the noise background.
Revised: The total number of revealed pixels was equal for every image at a given level of image masking, and the revealed pixels were centered against the noise background.

Original: Each participant saw only one masked version of each object image.
Revised: Each participant saw only one masked version of each object image.

Original: Psychophysics experiment.
Revised: Psychophysics experiment.

Original: Participants were instructed to categorize images in the psychophysics dataset as animals or non-animals as quickly and accurately as possible.
Revised: Participants were instructed to categorize images in the psychophysics dataset as animals or non-animals as quickly and accurately as possible.

Original: Each experimental trial consisted of the following sequence of events overlaid onto a white background (SI Fig.
Revised: Each experimental trial consisted of the following sequence of events overlaid onto a white background (SI Fig.

Original: 1): ( i ) a fixation cross displayed for a variable time (1,100–1,600ms); ( ii ) an image for 400ms; ( iii ) an additional 150ms of response time.
Revised: 1): ( i ) a fixation cross displayed for a variable time (1,100–1,600ms); ( ii ) an image for 400ms; ( iii ) an additional 150ms of response time.

Original: In other words, the experiment forced participants to perform rapid object categorization.
Revised: In other words, the experiment forced participants to perform rapid object categorization.

Original: They were given a total of 550ms to view an image and press a button to indicate its category (feedback was provided on trials in which responses were not provided within this time limit).
Revised: They were given a total of 550ms to view an image and press a button to indicate its category (feedback was provided on trials in which responses were not provided within this time limit).

Original: Images were sized at 256 x 256 pixel resolution, which is equivalent to a stimulus size approximately between 5 – 11 degrees of visual angle across a likely range of possible display and seating setups we expect participants used for the experiment.
Revised: Images were sized at 256 x 256 pixel resolution, which is equivalent to a stimulus size approximately between 5 – 11 degrees of visual angle across a likely range of possible display and seating setups we expect participants used for the experiment.

Original: Similar paradigms and timing parameters have been shown to capture pre-attentive visual system processing [31,49–51].
Revised: Similar paradigms and timing parameters have been shown to capture pre-attentive visual system processing [31,49–51].

Original: Participants provided informed consent electronically and were compensated $3.00 for their time ( ∼ 10–15 min; approximately $15.00/hr).
Revised: Participants provided informed consent electronically and were compensated $3.00 for their time ( ∼ 10–15 min; approximately $15.00/hr).

Original: Models.
Revised: Models.

Original: We compared humans with 84 different DNNs representing the variety of approaches used in the field today: 50 CNNs trained on ImageNet [1, 48, 52–63, 63–72], 6 CNNs trained on other datasets in addition to ImageNet (which we refer to as “CNN extra data”)
Revised: We compared humans with 84 different DNNs representing the variety of approaches used in the field today: 50 CNNs trained on ImageNet [1, 48, 52–63, 63–72], 6 CNNs trained on other datasets in addition to ImageNet (which we refer to as “CNN extra data”)

Original: [1,65,73], 10 vision transformers [74–78], 6 CNNs trained with self-supervision [79, 80], and 13 models trained for robustness to noise or adversarial examples [81,82].
Revised: [1,65,73], 10 vision transformers [74–78], 6 CNNs trained with self-supervision [79, 80], and 13 models trained for robustness to noise or adversarial examples [81,82].

Original: We used pretrained weights for each of these models supplied by their authors, with a variety of licenses (detailed in SI §2), implemented in
Revised: We used pretrained weights for each of these models supplied by their authors, with a variety of licenses (detailed in SI §2), implemented in

Original: Tensorflow 2.0, Keras, or PyTorch.
Revised: Tensorflow 2.0, Keras, or PyTorch.

Original:  Results 4.1 Where are diagnostic object features for humans and DNNs?
Revised:  Results 4.1 Where are diagnostic object features for humans and DNNs?

Original: To systematically compare the visual strategies of object recognition for humans and DNNs on
Revised: To systematically compare the visual strategies of object recognition for humans and DNNs on

Original: ImageNet, we first turned to the ClickMe dataset of feature importance maps [20].
Revised: ImageNet, we first turned to the ClickMe dataset of feature importance maps [20].

Original: In order to derive comparable feature importance maps for DNNs, we needed a method that could be efficiently and consistently applied to each of the 84 DNNs we tested without any idiosyncratic hyperparameters.
Revised: In order to derive comparable feature importance maps for DNNs, we needed a method that could be efficiently and consistently applied to each of the 84 DNNs we tested without any idiosyncratic hyperparameters.

Original: This led us to choose a classic method for explainable artificial intelligence, image feature saliency [83].
Revised: This led us to choose a classic method for explainable artificial intelligence, image feature saliency [83].

Original: We prepared human feature importance maps from ClickMe by taking the average importance map produced by humans for every image that also appeared in ImageNet validation.
Revised: We prepared human feature importance maps from ClickMe by taking the average importance map produced by humans for every image that also appeared in ImageNet validation.

Original: We then used
Revised: We then used

Original: Spearman’s rank-correlation to measure the similarity between human feature maps and DNN feature maps for each image [49].
Revised: Spearman’s rank-correlation to measure the similarity between human feature maps and DNN feature maps for each image [49].

Original: We also computed the inter-rater alignment of human feature importance maps as the mean split-half correlation across 1000 random splits of the participant pool ( ρ = 0 . 66 ).
Revised: We also computed the inter-rater alignment of human feature importance maps as the mean split-half correlation across 1000 random splits of the participant pool ( ρ = 0 . 66 ).

Original: We then normalized each human-DNN correlation by this score [20].
Revised: We then normalized each human-DNN correlation by this score [20].

Original: There were dramatic qualitative differences between the features selected by humans and DNNs on
Revised: There were dramatic qualitative differences between the features selected by humans and DNNs on

Original: ImageNet.
Revised: ImageNet.

Original: In general, humans selected less context and focused more on object parts: for animals, parts of their faces; for non-animals, parts that enable their usage, like the spade of a shovel (see
Revised: In general, humans selected less context and focused more on object parts: for animals, parts of their faces; for non-animals, parts that enable their usage, like the spade of a shovel (see

Original: Fig.
Revised: Fig.

Original: 2 and SI Fig.
Revised: 2 and SI Fig.

Original: The DNN that was most aligned with humans, the DenseNet121, was still only 38% aligned with humans (Fig. 3).
Revised: The DNN that was most aligned with humans, the DenseNet121, was still only 38% aligned with humans (Fig. 3).

Original: Plotting the relationship between DNNs’ top-1 accuracy on ImageNet with their human alignment revealed a striking trade-off: as the accuracy of DNNs has improved beyond DenseNet121,
Revised: Plotting the relationship between DNNs’ top-1 accuracy on ImageNet with their human alignment revealed a striking trade-off: as the accuracy of DNNs has improved beyond DenseNet121,

Original: their
Revised: their

Original: [Figure]
Revised: [Figure]

Original: S n a k e
Revised: S n a k e

Original: B e a r
Revised: B e a r

Original: H a r e
Revised: H a r e

Original: L e o p a r d
Revised: L e o p a r d

Original: B a ll
Revised: B a ll

Original: SimCLR alignment with humans has worsened (Fig. 3).
Revised: SimCLR alignment with humans has worsened (Fig. 3).

Original: For example, consider the ConvNext [1], which achieved the best top-1 accuracy in our experiments (85.8%), was only 22% aligned with humans – equivalent to the alignment of the BagNet33 [68] (63% top-1 accuracy).
Revised: For example, consider the ConvNext [1], which achieved the best top-1 accuracy in our experiments (85.8%), was only 22% aligned with humans – equivalent to the alignment of the BagNet33 [68] (63% top-1 accuracy).

Original: As an additional control, we computed the similarity between the average ClickMe map, which exhibits a center bias [84,85] (SI
Revised: As an additional control, we computed the similarity between the average ClickMe map, which exhibits a center bias [84,85] (SI

Original: Fig. 5), and each individual ClickMe map.
Revised: Fig. 5), and each individual ClickMe map.

Original: This center-bias control was only outperformed by 42/
Revised: This center-bias control was only outperformed by 42/

Original: CNNs we tested ( † in Fig. 3).
Revised: CNNs we tested ( † in Fig. 3).

Original: Overall, we observe that human and DNN alignment has considerably worsened since the introduction of these two models.
Revised: Overall, we observe that human and DNN alignment has considerably worsened since the introduction of these two models.

Original: The neural harmonizer.
Revised: The neural harmonizer.

Original: While scaling DNNs has immensely helped performance on popular benchmark tasks, there are still fundamental differences in the architectures of DNNs and the human visual system [37] which could part of the reason to blame for poor alignment.
Revised: While scaling DNNs has immensely helped performance on popular benchmark tasks, there are still fundamental differences in the architectures of DNNs and the human visual system [37] which could part of the reason to blame for poor alignment.

Original: While introducing biological constraints into DNNs could help this problem, there is plenty of evidence that doing so would hurt benchmark performance and require bespoke development for every different architecture [86–88].
Revised: While introducing biological constraints into DNNs could help this problem, there is plenty of evidence that doing so would hurt benchmark performance and require bespoke development for every different architecture [86–88].

Original: Is it possible to align a DNN’s visual strategies with humans without hurting its performance?
Revised: Is it possible to align a DNN’s visual strategies with humans without hurting its performance?

Original: Such a general-purpose method for aligning human and DNN visual strategies should satisfy the following criteria: ( i )
Revised: Such a general-purpose method for aligning human and DNN visual strategies should satisfy the following criteria: ( i )

Original: The method should work with any fully-differentiable network architecture.
Revised: The method should work with any fully-differentiable network architecture.

Original: ( ii ) It should not present optimization issues that interfere with learning to solve a task, and the task-accuracy of a model trained with the method should not be worse than a model trained without the method.
Revised: ( ii ) It should not present optimization issues that interfere with learning to solve a task, and the task-accuracy of a model trained with the method should not be worse than a model trained without the method.

Original: We created the neural harmonizer to satisfy these criteria.
Revised: We created the neural harmonizer to satisfy these criteria.

Original: Let us consider a supervised categorization problem with an input space, X an output space Y ⊆
Revised: Let us consider a supervised categorization problem with an input space, X an output space Y ⊆

Original: R c and a predictor function f
Revised: R c and a predictor function f

Original: θ : X → Y parameterized by θ , which maps an input vector x ∈ X to an output f θ ( x ) .
Revised: θ : X → Y parameterized by θ , which maps an input vector x ∈ X to an output f θ ( x ) .

Original: We denote g : F × X → X an explanation functional that, given a predictor f θ ∈ F and an input, returns a feature importance map ϕ =
Revised: We denote g : F × X → X an explanation functional that, given a predictor f θ ∈ F and an input, returns a feature importance map ϕ =

Original: g ( f θ , x ) .
Revised: g ( f θ , x ) .

Original: Here, we focus on DNN saliency g ( f θ , x ) ≜ ∇ x f θ ( x ) as our method for computing feature importance in DNNs, but the method can in principle work with any differentiable network explanation method.
Revised: Here, we focus on DNN saliency g ( f θ , x ) ≜ ∇ x f θ ( x ) as our method for computing feature importance in DNNs, but the method can in principle work with any differentiable network explanation method.

Original: To satisfy criterion ( i ), the neural harmonizer introduces a differentiable loss that will enforce alignment across feature importance map scales from any neural network.
Revised: To satisfy criterion ( i ), the neural harmonizer introduces a differentiable loss that will enforce alignment across feature importance map scales from any neural network.

Original: Let P i ( . ) be the function mapping a feature importance map ϕ to it is representation in the N levels of a Gaussian pyramid, with i ∈ { 1 , ..., N } .
Revised: Let P i ( . ) be the function mapping a feature importance map ϕ to it is representation in the N levels of a Gaussian pyramid, with i ∈ { 1 , ..., N } .

Original: The function P i ( ϕ ) is computed by downsampling P i − 1 ( ϕ ) using a Gaussian
Revised: The function P i ( ϕ ) is computed by downsampling P i − 1 ( ϕ ) using a Gaussian

Original: [Figure]
Revised: [Figure]

Original: CNNTransformerSelf
Revised: CNNTransformerSelf

Original: [Figure]
Revised: [Figure]

Original: H u m a n f e a t u r e a li g n m e n t ( % ) † kernel, with P 1 ( ϕ ) = ϕ .
Revised: H u m a n f e a t u r e a li g n m e n t ( % ) † kernel, with P 1 ( ϕ ) = ϕ .

Original: We then seek to minimize (cid:80) Ni ||P
Revised: We then seek to minimize (cid:80) Ni ||P

Original: i ( g ( f θ , x ))
Revised: i ( g ( f θ , x ))

Original: − P i ( ϕ ) || 2 , which will align feature importance maps between humans and DNNs at every scale of the pyramid.
Revised: − P i ( ϕ ) || 2 , which will align feature importance maps between humans and DNNs at every scale of the pyramid.

Original: To satisfy criterion ( ii ), the neural harmonizer should work well with training routines designed for large-scale computer vision challenges like ImageNet.
Revised: To satisfy criterion ( ii ), the neural harmonizer should work well with training routines designed for large-scale computer vision challenges like ImageNet.

Original: This means that the neural harmonizer loss must avoid optimization issues at scale.
Revised: This means that the neural harmonizer loss must avoid optimization issues at scale.

Original: To do this, we need a way of comparing feature importance maps between humans and DNNs that is invariant to the norm of either map.
Revised: To do this, we need a way of comparing feature importance maps between humans and DNNs that is invariant to the norm of either map.

Original: We therefore standardize feature importance maps from humans and DNNs before comparing them, and only measure alignment on the most important areas of the image for each observer.
Revised: We therefore standardize feature importance maps from humans and DNNs before comparing them, and only measure alignment on the most important areas of the image for each observer.

Original: Formally, let z ( . ) be a standardization function over feature importance maps that takes the mean and standard deviation computed for each map ϕ such that z ( ϕ ) has 0 activation on average and unit standard deviation.
Revised: Formally, let z ( . ) be a standardization function over feature importance maps that takes the mean and standard deviation computed for each map ϕ such that z ( ϕ ) has 0 activation on average and unit standard deviation.

Original: To focus alignment on important regions, let z ( ϕ ) + denote the positive part of the standardized explanation z ( ϕ ) .
Revised: To focus alignment on important regions, let z ( ϕ ) + denote the positive part of the standardized explanation z ( ϕ ) .

Original: Finally, we include a task loss, the familiar cross entropy objective, to yield the complete neural harmonization loss and train models that are at least as accurate as those trained without harmonization:
Revised: Finally, we include a task loss, the familiar cross entropy objective, to yield the complete neural harmonization loss and train models that are at least as accurate as those trained without harmonization:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: n
Revised: m

Revised: a n f e a t u

Original: [Figure]
Revised: [Figure]

Original: a
Revised: li

Original: [Figure]
Revised: [Figure]

Original: CNNTransformerSelf -supervised
Revised: CNNTransformerSelf -

Original: [Figure]
Revised: [Figure]

Original: Training.
Revised: Training.

Original: We trained four different DNNs with the neural harmonizer:
Revised: We trained four different DNNs with the neural harmonizer:

Original: VGG16, ViT, ResNet50, and
Revised: VGG16, ViT, ResNet50, and

Original: EfficientNetB0.
Revised: EfficientNetB0.

Original: These models were selected because they are popular convolutional and transformer networks with open-source architectures that are straightforward to train and also sit near the boundary of the trade-off between DNN performance and alignment with humans.
Revised: These models were selected because they are popular convolutional and transformer networks with open-source architectures that are straightforward to train and also sit near the boundary of the trade-off between DNN performance and alignment with humans.

Original: Models were trained using the neural harmonizer to optimize categorization performance on ImageNet and feature importance map alignment with human data from ClickMe .
Revised: Models were trained using the neural harmonizer to optimize categorization performance on ImageNet and feature importance map alignment with human data from ClickMe .

Original: We trained models on all images in the ImageNet training set, but because ClickMe only contains human feature importance maps for a portion of those images, we computed the categorization loss but not the neural harmonizer loss for images without importance maps.
Revised: We trained models on all images in the ImageNet training set, but because ClickMe only contains human feature importance maps for a portion of those images, we computed the categorization loss but not the neural harmonizer loss for images without importance maps.

Original: Models were trained using 8 cores V4 TPUs on the Google Cloud Platform, and training lasted approximately one day.
Revised: Models were trained using 8 cores V4 TPUs on the Google Cloud Platform, and training lasted approximately one day.

Original: Models were trained with an augmented ResNet training recipe (built from https://github.com/tensorflow/tpu/ ).
Revised: Models were trained with an augmented ResNet training recipe (built from https://github.com/tensorflow/tpu/ ).

Original: Models were optimized with SGD and momentum over batches of 512 images, a learning rate of 0 . 3 , and label smoothing [89].
Revised: Models were optimized with SGD and momentum over batches of 512 images, a learning rate of 0 . 3 , and label smoothing [89].

Original: Images were augmented with random left-right flips and mixup [90].
Revised: Images were augmented with random left-right flips and mixup [90].

Original: The learning rate was adjusted over the course of training with a schedule that began with an initial warm-up period of 5 epochs and then decaying according to a cosine function over 90 epochs, with decay at step 30, 50 and 80.
Revised: The learning rate was adjusted over the course of training with a schedule that began with an initial warm-up period of 5 epochs and then decaying according to a cosine function over 90 epochs, with decay at step 30, 50 and 80.

Original: We validated that a ResNet50 and VGG16 trained with these hyperparameters and schedule using standard cross-entropy (but not the neural harmonizer) matched published performance.
Revised: We validated that a ResNet50 and VGG16 trained with these hyperparameters and schedule using standard cross-entropy (but not the neural harmonizer) matched published performance.

Original: The neural harmonizer aligns human and DNN visual strategies.
Revised: The neural harmonizer aligns human and DNN visual strategies.

Original: We found that harmonized models broke the trade-off between ImageNet accuracy and model alignment with ClickMe human feature importance maps (Fig. 3).
Revised: We found that harmonized models broke the trade-off between ImageNet accuracy and model alignment with ClickMe human feature importance maps (Fig. 3).

Original: Harmonized models were significantly more aligned with feature importance maps and also performed better on ImageNet.
Revised: Harmonized models were significantly more aligned with feature importance maps and also performed better on ImageNet.

Original: The changes in where harmonized models find important features in images were dramatic: a harmonized ViT had feature importance maps that are far less reliant on context (Fig. 2) and approximately 150% more aligned with humans (Fig. 3;
Revised: The changes in where harmonized models find important features in images were dramatic: a harmonized ViT had feature importance maps that are far less reliant on context (Fig. 2) and approximately 150% more aligned with humans (Fig. 3;

Original: Top
Revised: Top

Original: [Figure]
Revised: [Figure]

Original: 1 ImageNet Accuracy (%)  80         - l o g ( m e a n s q u a r e d e rr o r )
Revised: 1 ImageNet Accuracy (%)  80         - l o g ( m e a n s q u a r e d e rr o r )

Original: 100 I m p o r t a n t hu
Revised: 100 I m p o r t a n t hu

Original: m
Revised: m

Original: a n f e a t u r e s ( % )
Revised: a n f e a t u r e s ( % )

Original: Human features revealed (%)
Revised: Human features revealed (%)

Original: [Figure]
Revised: [Figure]

Original: o r m a li
Revised: o r m a li

Original: z
Revised: z

Original: e
Revised: e

Original: d p e r f o r
Revised: d p e r f o r

Original: [Figure]
Revised: [Figure]

Original: a n c e
Revised: a n c e

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ViT goes from 28.7% to 72.6% alignment after harmonization).
Revised: ViT goes from 28.7% to 72.6% alignment after harmonization).

Original: The same model also performed 4% better in top-1 accuracy without any changes to its architecture.
Revised: The same model also performed 4% better in top-1 accuracy without any changes to its architecture.

Original: Similar improvements were found for the harmonized VGG16 and ResNet50.
Revised: Similar improvements were found for the harmonized VGG16 and ResNet50.

Original: While the EfficientNetB0 had only a minimal improvement in accuracy, it too exhibited a large boost in human feature alignment.
Revised: While the EfficientNetB0 had only a minimal improvement in accuracy, it too exhibited a large boost in human feature alignment.

Original: Clicktionary.
Revised: Clicktionary.

Original: To test if the trade-off between DNN ImageNet accuracy and alignment with humans is a general phenomenon we next turned to Clicktionary [22].
Revised: To test if the trade-off between DNN ImageNet accuracy and alignment with humans is a general phenomenon we next turned to Clicktionary [22].

Original: Indeed, we observed a similar trade-off on this dataset as we found for ClickMe : alignment with human feature importance from Clicktionary has worsened as DNN accuracy has improved on ImageNet (Fig. 4).
Revised: Indeed, we observed a similar trade-off on this dataset as we found for ClickMe : alignment with human feature importance from Clicktionary has worsened as DNN accuracy has improved on ImageNet (Fig. 4).

Original: As with ClickMe , harmonized
Revised: As with ClickMe , harmonized

Original: DNNs shift the accuracy-alignment trade-off on this dataset.
Revised: DNNs shift the accuracy-alignment trade-off on this dataset.

Original: 4.2 How do humans and DNNs integrate diagnostic object features into decisions?
Revised: 4.2 How do humans and DNNs integrate diagnostic object features into decisions?

Original: The trade-off we discovered between DNN accuracy on ImageNet and alignment with human visual feature importance suggests that the two use different visual strategies for object classification.
Revised: The trade-off we discovered between DNN accuracy on ImageNet and alignment with human visual feature importance suggests that the two use different visual strategies for object classification.

Original: However, there is potential for an even deeper problem.
Revised: However, there is potential for an even deeper problem.

Original: Even if two observers deem the same regions of an image as important for recognizing it, there is no guarantee that they use the selected features in the same way to render their decisions.
Revised: Even if two observers deem the same regions of an image as important for recognizing it, there is no guarantee that they use the selected features in the same way to render their decisions.

Original: We posit that if two observers have aligned visual strategies, the will agree on both where important features are in an image and how they use those features for decisions.
Revised: We posit that if two observers have aligned visual strategies, the will agree on both where important features are in an image and how they use those features for decisions.

Original: We developed a psychophysics experiment to measure how different humans use features in ImageNet images to recognize objects.
Revised: We developed a psychophysics experiment to measure how different humans use features in ImageNet images to recognize objects.

Original: Participants viewed versions of these images where only a proportion of the features that were deemed most important in the Clicktionary game were visible (Fig. 5a).
Revised: Participants viewed versions of these images where only a proportion of the features that were deemed most important in the Clicktionary game were visible (Fig. 5a).

Original: Participants had to accurately detect whether or not the image contained an animal within 550ms, which forced them to rely on feedforward processing as much as possible [33].
Revised: Participants had to accurately detect whether or not the image contained an animal within 550ms, which forced them to rely on feedforward processing as much as possible [33].

Original: Each of theimages we used were shown to a single participant only once.
Revised: Each of theimages we used were shown to a single participant only once.

Original: We accumulated responses from all participants to construct decision curves that showed how accurately the average human converted any given proportion of image features into an object decision.
Revised: We accumulated responses from all participants to construct decision curves that showed how accurately the average human converted any given proportion of image features into an object decision.

Original: We performed the same experiment on DNNs as we did on humans, recording animal vs. non-animal decisions according to whether or not the most probable category in the model’s 1000-category output was an animal.
Revised: We performed the same experiment on DNNs as we did on humans, recording animal vs. non-animal decisions according to whether or not the most probable category in the model’s 1000-category output was an animal.

Original: Because the experiment was speeded, humans did not achieve perfect accuracy.
Revised: Because the experiment was speeded, humans did not achieve perfect accuracy.

Original: Thus, we normalized performance for humans and DNNs to compare the rate at which each integrated features into accurate decisions.
Revised: Thus, we normalized performance for humans and DNNs to compare the rate at which each integrated features into accurate decisions.

Original: We discovered a similar trade-off between ImageNet accuracy and alignment with human visual decision making in this experiment as we did in ClickMe and Clicktionary (Fig. 5b).
Revised: We discovered a similar trade-off between ImageNet accuracy and alignment with human visual decision making in this experiment as we did in ClickMe and Clicktionary (Fig. 5b).

Original: Indeed, the model that was most aligned with human decision-making – the BagNet33 [68] – only achieved 63.0% accuracy on ImageNet.
Revised: Indeed, the model that was most aligned with human decision-making – the BagNet33 [68] – only achieved 63.0% accuracy on ImageNet.

Original: Surprisingly, harmonized models broke this trend, particularly the harmonized
Revised: Surprisingly, harmonized models broke this trend, particularly the harmonized

Original: ViT (Fig.
Revised: ViT (Fig.

Original: 5b, top-right), despite no explicit constraints in that procedure which forced consistent decision-making with humans.
Revised: 5b, top-right), despite no explicit constraints in that procedure which forced consistent decision-making with humans.

Original: In contrast, an unharmonized ViT integrates visual information into accurate decisions less efficiently than humans or harmonized models (Fig. 5c).
Revised: In contrast, an unharmonized ViT integrates visual information into accurate decisions less efficiently than humans or harmonized models (Fig. 5c).

Original:  Conclusion
Revised:  Conclusion

Original: Models that reliably categorize objects like humans do would shift the paradigms of the cognitive sciences and artificial intelligence.
Revised: Models that reliably categorize objects like humans do would shift the paradigms of the cognitive sciences and artificial intelligence.

Original: But despite continuous progress over the past decade on the
Revised: But despite continuous progress over the past decade on the

Original: ImageNet benchmark, DNNs are becoming worse models of human vision.
Revised: ImageNet benchmark, DNNs are becoming worse models of human vision.

Original: Our solution to this problem, the neural harmonizer, can be applied to any DNN to align their visual strategies with humans and even improve performance.
Revised: Our solution to this problem, the neural harmonizer, can be applied to any DNN to align their visual strategies with humans and even improve performance.

Original: We observed the greatest benefit of harmonization on the visual transformer, the ViT.
Revised: We observed the greatest benefit of harmonization on the visual transformer, the ViT.

Original: This finding is particularly surprising given that transformers eschew the locality bias of convolutional neural networks that has helped them become the new standard for modeling human vision and cognition [37].
Revised: This finding is particularly surprising given that transformers eschew the locality bias of convolutional neural networks that has helped them become the new standard for modeling human vision and cognition [37].

Original: Thus, we suspect that the neural harmonizer is especially well-suited for large-scale training on lowinductive bias models, like transformers.
Revised: Thus, we suspect that the neural harmonizer is especially well-suited for large-scale training on lowinductive bias models, like transformers.

Original: We also hypothesize that the improvements in human alignment provided by the neural harmonizer will yield a variety of downstream benefits for a model like the ViT, including better predictions of perceptual similarity, stimulus-evoked neural responses, and even performance on visual reasoning tasks.
Revised: We also hypothesize that the improvements in human alignment provided by the neural harmonizer will yield a variety of downstream benefits for a model like the ViT, including better predictions of perceptual similarity, stimulus-evoked neural responses, and even performance on visual reasoning tasks.

Original: We leave these analyses for future work.
Revised: We leave these analyses for future work.

Original: The field of computer vision today is following Sutton’s prescient lesson: benchmark tasks can be scaling architectural capacity and the size of training data.
Revised: The field of computer vision today is following Sutton’s prescient lesson: benchmark tasks can be scaling architectural capacity and the size of training data.

Original: However, as we have demonstrated here, these scaling laws are exchanging performance for alignment with human perception.
Revised: However, as we have demonstrated here, these scaling laws are exchanging performance for alignment with human perception.

Original: We encourage the field to re-analyze the costs and benefits of this exchange, particularly in light of the growing concerns about DNNs leveraging shortcuts and dataset biases to achieve high performance [19].
Revised: We encourage the field to re-analyze the costs and benefits of this exchange, particularly in light of the growing concerns about DNNs leveraging shortcuts and dataset biases to achieve high performance [19].

Original: Alignment with human vision need not be exchanged with performance if DNNs are harmonized.
Revised: Alignment with human vision need not be exchanged with performance if DNNs are harmonized.

Original: Our codebase ( https://serre-lab.github.io/Harmonization/ ) can be used to incorporate the neural harmonizer into any DNN created and measure its alignment with humans on the datasets we describe in this paper.
Revised: Our codebase ( https://serre-lab.github.io/Harmonization/ ) can be used to incorporate the neural harmonizer into any DNN created and measure its alignment with humans on the datasets we describe in this paper.

Original: Limitations.
Revised: Limitations.

Original: One possible explanation for the misalignment between DNNs and humans that we observe is that recent DNNs have achieved superhuman accuracy on ImageNet.
Revised: One possible explanation for the misalignment between DNNs and humans that we observe is that recent DNNs have achieved superhuman accuracy on ImageNet.

Original: Superhuman
Revised: Superhuman

Original: DNNs have been described in biomedical applications [92,93] where there is definitive biological ground-truth labels, but ImageNet labels are noisy, making it unclear if such an achievement is laudable ( https://labelerrors.com/ ).
Revised: DNNs have been described in biomedical applications [92,93] where there is definitive biological ground-truth labels, but ImageNet labels are noisy, making it unclear if such an achievement is laudable ( https://labelerrors.com/ ).

Original: Thus, an equally likely explanation is that the continued improvements of DNNs at least partially reflect their exploitation of shortcuts in ImageNet [19].
Revised: Thus, an equally likely explanation is that the continued improvements of DNNs at least partially reflect their exploitation of shortcuts in ImageNet [19].

Original: The scope of our work is also limited in that it focuses on object recognition in ImageNet.
Revised: The scope of our work is also limited in that it focuses on object recognition in ImageNet.

Original: It is possible that models trained on other tasks, such as segmentation, may be more aligned with humans.
Revised: It is possible that models trained on other tasks, such as segmentation, may be more aligned with humans.

Original: Finally, our modeling efforts were hamstrung for the largest-scale models in existence.
Revised: Finally, our modeling efforts were hamstrung for the largest-scale models in existence.

Original: Our work does not answer how much harmonization would help a model like CLIP because of the massive investment needed to train it.
Revised: Our work does not answer how much harmonization would help a model like CLIP because of the massive investment needed to train it.

Original: The neural harmonizer can be applied to CLIP but it is possible that more ClickMe human feature importance maps are needed for successful harmonization.
Revised: The neural harmonizer can be applied to CLIP but it is possible that more ClickMe human feature importance maps are needed for successful harmonization.

Original: Broader impacts.
Revised: Broader impacts.

Original: A persistent issue in the field of artificial intelligence is the tendency of models to exploit dataset biases.
Revised: A persistent issue in the field of artificial intelligence is the tendency of models to exploit dataset biases.

Original: A central theme of our work is that there are facets of human perception that are not captured by DNNs, particularly those which follow the scaling laws which have been so embraced by industry leaders.
Revised: A central theme of our work is that there are facets of human perception that are not captured by DNNs, particularly those which follow the scaling laws which have been so embraced by industry leaders.

Original: Forcing DNNs to rely on similar visual strategies as humans could represent a scalable path forward to correcting the insidious biases which have assailed under-constrained models of artificial intelligence.
Revised: Forcing DNNs to rely on similar visual strategies as humans could represent a scalable path forward to correcting the insidious biases which have assailed under-constrained models of artificial intelligence.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work was supported by ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481),
Revised: This work was supported by ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481),

Original: DARPA (D19AC00015), NIH/NINDS (R21 NS 112743), and the ANR-3IA Artificial and Natural
Revised: DARPA (D19AC00015), NIH/NINDS (R21 NS 112743), and the ANR-3IA Artificial and Natural

Original: Intelligence Toulouse Institute (ANR-19-PI3A-0004).
Revised: Intelligence Toulouse Institute (ANR-19-PI3A-0004).

Original: Additional support provided by the Carney
Revised: Additional support provided by the Carney

Original: Institute for Brain Science and the Center for Computation and Visualization (CCV).
Revised: Institute for Brain Science and the Center for Computation and Visualization (CCV).

Original: We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant
Revised: We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant

Original: S10OD
Revised: S10OD

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Methods.
Revised: [Yes] See Methods.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Discussion.
Revised: [Yes] See Discussion.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See Discussion (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Discussion (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See Methods (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [Yes] See Methods (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] See Methods 5.
Revised: [Yes] See Methods 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Yes] See Methods
Revised: [Yes] See Methods