Original: Aligning human and machine vision
Revised: Harmonizing deep neural network strategies for object recognition with humans

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The many successes of deep neural networks (DNNs) over the past decade havelargely been driven by computational scale rather than insights from biologicalintelligence.
Revised: The many successes of deep neural networks (DNNs) over the past decade havelargely been driven by computational scale rather than insights from biologicalintelligence.

Original: Here, we explore if these trends have also carried concomitant improvements in explaining visual strategies underlying human object recognition.
Revised: Here, we explore if these trends have also carried concomitant improvements in explaining visual strategies underlying human object recognition.

Original: We do this by comparing two related but distinct properties of visual strategies inhumans and DNNs: where they believe important visual features are in images andhow
Revised: We do this by comparing two related but distinct properties of visual strategies inhumans and DNNs: where they believe important visual features are in images andhow

Original: they use those features to categorize objects.
Revised: they use those features to categorize objects.

Original: Across 85 different DNNs andthree independent datasets measuring human visual strategies on ImageNet, wefind a trade-off between DNN top-1 categorization accuracy and their alignmentwith humans.
Revised: Across 85 different DNNs andthree independent datasets measuring human visual strategies on ImageNet, wefind a trade-off between DNN top-1 categorization accuracy and their alignmentwith humans.

Original: State-of-the-art DNNs are progressively becoming less aligned withhumans .
Revised: State-of-the-art DNNs are progressively becoming less aligned withhumans .

Original: We rectify this growing issue by introducing the neural harmonizer: ageneral-purpose training routine that aligns DNN and human visual strategies whileimproving object classification performance.
Revised: We rectify this growing issue by introducing the neural harmonizer: ageneral-purpose training routine that aligns DNN and human visual strategieswhile improving object classification performance.

Original: Our work represents the first systematic demonstration that the scaling laws [1–3] that are guiding DNN developmentstoday have also produced worse models of human vision.
Revised: Our work represents the firstsystematic demonstration that the scaling laws [1–3] that are guiding DNN developments today have also produced worse models of human vision.

Original: We release our code anddata to help the field build more human-like DNNs.
Revised: We release ourcode and data at https://tinyurl.com/metapred to help the field build morehuman-like DNNs.

Original:  Introduction
Revised:  Introduction

Original: In his bitter lesson [4], Rich Sutton wrote, “The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective andby a large margin.”
Revised: Rich Sutton stated [4] that the bitter lesson “from 70 years of AI research is that general methodsthat leverage computation are ultimately the most effective, and by a large margin.”

Original: Deep learning has been the standard approach to object classification problemsever since the paradigm shifting success of AlexNet [5] on the ImageNet [6] benchmark a decadeago.
Revised: Deep learning has been the standard approach to object classification problems ever since the paradigm shiftingsuccess of AlexNet [5] on the ImageNet [6] benchmark a decade ago.

Original: As deep neural network (DNN) performance has continued to improve in the interveningyears, Sutton’s lesson has become more fitting than ever, with recent networks rivaling and likelyoutperforming humans on the benchmark [7] through brute-force computational scale: increasing thenumber of network parameters and number of images used for training orders-of-magnitude beyond AlexNet [1–3].
Revised: As deep neural network (DNN)performance has continued to improve in the intervening years, Sutton’s lesson has become morefitting than ever, with recent networks rivaling and likely outperforming humans on the benchmark [7]through brute-force computational scale: increasing the number of network parameters and number ofimages used for training orders-of-magnitude beyond AlexNet [1–3].

Original: While the successes of so-called “scaling laws” are undeniable, this singular focus onperformance in the field has side-stepped an equally important question that will govern the utility ofobject recognition models for brain sciences and industry applications alike: are the visual strategiesof DNNs aligned with human perception?
Revised: While the successes of so-called“scaling laws” are undeniable, this singular focus on performance in the field has side-stepped anequally important question that will govern the utility of object recognition models for brain sciencesand industry applications alike: are the visual strategies of DNNs aligned with human perception?

Original: The visual strategies that mediate object recognition in humans can be decomposed into two relatedbut distinct processes: identifying where the important features for object recognition are in a scene,and determining how to integrate the selected features into a categorical decision [8,9].
Revised: The visual strategies that mediate object recognition in humans can be decomposed into two relatedbut distinct processes: identifying where the important features for object recognition are in a scene,and determining how to integrate the selected features into a categorical decision [8,9].

Original: It has beenknown for nearly a century [10–13] that different humans attend to similar locations when askedto find and recognize objects.
Revised: It has beenknown for nearly a century [10–13] that different humans attend to similar locations when askedto find and recognize objects.

Original: After selecting these important features, human observers are alsoconsistent in how they use those features to classify objects.
Revised: After selecting these important features, human observers are alsoconsistent in how they use those features to classify objects.

Original: The inclusion of a few pixels in an imagecan be the difference between recognizing an object or not [9,14].
Revised: The inclusion of a few pixels in an imagecan be the difference between recognizing an object or not [9,14].

Original: Has the past decade of DNN development produced any models that are aligned with these humanvisual strategies for object recognition?
Revised: Has the past decade of DNN development produced any models that are aligned with these humanvisual strategies for object recognition?

Original: Such a model could transform cognitive science by support- ing a better mechanistic understanding of how vision works.
Revised: Such a model could transform cognitive science by supporting a better mechanistic understanding of how vision works.

Original: More human-like models of objectrecognition would also resolve the problems with predictablity and interpretablity of DNNs [15–18],and control their alarming tendency to rely on “shortcuts” and dataset biases to perform well ontasks [19].
Revised: More human-like models of objectrecognition would also resolve the problems with predictablity and interpretablity of DNNs [15–18],and control their alarming tendency to rely on “shortcuts” and dataset biases to perform well ontasks [19].

Original: In this work, we perform the first large-scale and systematic comparison of the visualstrategies of DNNs and humans for object recognition on ImageNet.
Revised: In this work, we perform the first large-scale and systematic comparison of the visualstrategies of DNNs and humans for object recognition on ImageNet.

Original: Contributions.
Revised: Contributions.

Original: In order to compare human and DNN visual strategies, we first turn to the humanfeature importance maps collected by Linsley et al.
Revised: In order to compare human and DNN visual strategies, we first turn to the humanfeature importance maps collected by Linsley et al.

Original: Their datasets, ClickMe and Clicktionary ,contain maps of nearly 200,000 unique images in ImageNet that highlight the visual features humansbelieve are important for recognizing them.
Revised: Their datasets, ClickMe and Clicktionary ,contain maps of nearly 200,000 unique images in ImageNet that highlight the visual features humansbelieve are important for recognizing them.

Original: These datasets amount to a reverse inference on whereimportant visual features are in ImageNet images (Fig. 1).
Revised: These datasets amount to a reverse inference on whereimportant visual features are in ImageNet images (Fig. 1).

Original: We complement these datasets withnew psychophysics experiments that directly test how important visual features are used for objectrecognition (Fig. 1).
Revised: We complement these datasets withnew psychophysics experiments that directly test how important visual features are used for objectrecognition (Fig. 1).

Original: As DNN performance has increased on ImageNet, their alignment with humanvisual strategies captured in these datasets has worsened.
Revised: As DNN performance has increased on ImageNet, their alignment with humanvisual strategies captured in these datasets has worsened.

Original: This trade-off persists over 85 different
Revised: This trade-off persists over 85 different

Original: DNNs representing all popular model classes – from those trained for adversarial robustness to thosepushing the scaling laws in network capacity and training data.
Revised: DNNs representing all popular model classes – from those trained for adversarial robustness to thosepushing the scaling laws in network capacity and training data.

Original: To summarize our findings:
Revised: To summarize our findings:

Original: • The trade-off between DNN object recognition accuracy and alignment with human visual strategies replicates across three unique datasets: ClickMe [20], Clicktionary [21]
Revised: • The trade-off between DNN object recognition accuracy and alignment with human visual strategiesreplicates across three unique datasets: ClickMe [20], Clicktionary [21], and our psychophysics

Original: and our psychophysics

Original: experiments.
Revised: experiments.

Original: • We shift this trade-off with our neural harmonizer, a novel drop-in module for co-training any DNNto align with human visual strategies while also achieving high task accuracy.
Revised: • We shift this trade-off with our neural harmonizer, a novel drop-in module for co-training any DNN to align with human visual strategies while also achieving high task accuracy.

Original: DNNs trained withthe neural harmonizer learn visual strategies that are significantly more aligned with humans than
Revised: DNNs trained withthe neural harmonizer learn visual strategies that are significantly more aligned with humans than

Original: any other DNN we tested .
Revised: any other DNN we tested .

Original: • We release our data, code, and human psychophysics at https://tinyurl.com/metapred tohelp the field tackle the growing misalignment between DNNs and humans.
Revised: • We release our data, code, and human psychophysics at https://

Original: Related work
Revised: work

Original: Human feature preference Less More
Revised: Human feature preference Less More

Original: Where are the important features?
Revised: Where are the important features?

Original: ρ ( Human , DNN )
Revised: ρ ( Human , DNN )

Original: Human
Revised: Human

Original: DNN
Revised: DNN

Original: T a r ge t
Revised: T a r ge t

Original: D i s t r a c t
Revised: D i s t r a c t

Original: o r
Revised: o r

Original: Percent image revelation 1% 25% 63% 100% Full
Revised: Percent image revelation 1% 25% 63% 100% Full

Original: Processed feature importance map + +
Revised: Processed feature importance map + +

Original: Please respond faster!
Revised: Please respond faster!

Original: Pre-stimulus ﬁxation (1,100 – 1,600ms)
Revised: Pre-stimulus ﬁxation (1,100 – 1,600ms)

Original: Stimulus400ms
Revised: Stimulus400ms

Original: Extra response 150ms …
Revised: Extra response 150ms …

Original: if window
Revised: if window

Original: Is reached
Revised: Is reached

Original: Experiment trial
Revised: Experiment trial

Original: T a r ge t
Revised: T a r ge t

Original: D i s t r a c t
Revised: D i s t r a c t

Original: o r
Revised: o r

Original: Percent image revelation 1% 25% 63% 100%
Revised: Percent image revelation 1% 25% 63% 100%

Original: Full
Revised: Full

Original: Processed feature importance map + +
Revised: Processed feature importance map + +

Original: Please respond faster!
Revised: Please respond faster!

Original: Pre-stimulus ﬁxation (1,100 – 1,600ms)
Revised: Pre-stimulus ﬁxation (1,100 – 1,600ms)

Original: Stimulus400ms
Revised: Stimulus400ms

Original: Extra response 150ms …
Revised: Extra response 150ms …

Original: if window
Revised: if window

Original: Is reached
Revised: Is reached

Original: Experiment trial
Revised: Experiment trial

Original: T a r ge t
Revised: T a r ge t

Original: D i s t r a c t
Revised: D i s t r a c t

Original: o r
Revised: o r

Original: Percent image revelation 1% 25% 63% 100%
Revised: Percent image revelation 1% 25% 63% 100%

Original: Full
Revised: Full

Original: Processed feature importance map + +
Revised: Processed feature importance map + +

Original: Please respond faster!
Revised: Please respond faster!

Original: Pre-stimulus ﬁxation (1,100 – 1,600ms)
Revised: Pre-stimulus ﬁxation (1,100 – 1,600ms)

Original: Stimulus400ms
Revised: Stimulus400ms

Original: Extra response 150ms …
Revised: Extra response 150ms …

Original: if window
Revised: if window

Original: Is reached
Revised: Is reached

Original: Experiment trial
Revised: Experiment trial

Original: [Figure]
Revised: [Figure]

Original: \textit{Comparing visual strategies of humans and machines in object classiﬁcation.}
Revised: [Figure] on their ability to explain human vision.

Original: How are the discovered features used? explain counts evidence ter of humans on challengingvisual tasks, such as recognizing objects obscured by noise [23].
Revised: On the one hand, there isevidence that DNNs are becoming better models of humans on challengingvisual tasks, such as recognizing objects obscured by noise [23].

Original: On theother hand, there is a growing numberof reports that DNNs struggle to explain perceptual phenomena in humanvision like contextual illusions [24],perceptual grouping [19,25,26], andcategorical prototypes [27].
Revised: On theother hand, there is a growing numberof reports that DNNs struggle to explain perceptual phenomena in humanvision like contextual illusions [24],perceptual grouping [19,25,26], andcategorical prototypes [27].

Original: Othershave found differences between human attention data and DNN models of attention [20,28].
Revised: Othershave found differences between human attention data and DNN models of attention [20,28].

Original: Moreover, similarities between DNNs and neural representations of objects have become less clear overthe years.
Revised: Moreover, similarities between DNNs and neural representations of objects have become less clear overthe years.

Original: While the original theory was that model explanations of neural representations of objectimages improved alongside model classification accuracy [29], recent large-scale DNNs are worse atexplaining neural data than older ones with lower ImageNet accuracy [30].
Revised: While the original theory was that model explanations of neural representations of objectimages improved alongside model classification accuracy [29], recent large-scale DNNs are worse atexplaining neural data than older ones with lower ImageNet accuracy [30].

Original: What are the visual strategies underlying human object recognition?
Revised: What are the visual strategies underlying human object recognition?

Original: Ever since its inception, agoal of vision science has been to characterize the processes supporting human object recognition.
Revised: Ever since its inception, agoal of vision science has been to characterize the processes supporting human object recognition.

Original: It has been discovered that object recognition can be decomposed into different processing stagesthat emerge over time [8, 31–34], where the earliest stage is associated with processing throughfeedforward connections in the visual system, and the later stage is associated with processing throughfeedback connections.
Revised: It has been discovered that object recognition can be decomposed into different processing stagesthat emerge over time [8, 31–36], where the earliest stage is associated with processing throughfeedforward connections in the visual system, and the later stage is associated with processing through feedback connections.

Original: Since the DNNs used today rely on feedforward connections, it is likely thatthey are better models for that rapid feedforward phase of processing than the subsequent feedback phase [33, 35].
Revised: Since the DNNs used today rely on feedforward connections, it is likely thatthey are better models for that rapid feedforward phase of processing than the subsequent feedbackphase [33,37].

Original: Thus, to maximize the likelihood of alignment between DNN and human visualstrategies, our experiments described in §3 test visual strategies underlying rapid feedforward objectrecognition in humans.
Revised: To maximize the likelihood of alignment between DNN and human visual strategies,our experiments focus on the visual strategies of rapid feedforward object recognition in humans.

Original: Most closely related to our work, and specifically analyses of where important visual featuresare in images, are studies of “top-down” image saliency.
Revised: Most closely related to our work, and specifically analyses of where important visual featuresare in images, are studies of “top-down” image saliency.

Original: These involve asking participants tosearch for an object in an image, or specifically find the visual features needed to recognize anobject [10–13,20,22,36].
Revised: These involve asking participants tosearch for an object in an image, or specifically find the visual features needed to recognize anobject [10–13,20,22,38].

Original: In our work, we complement these descriptions of where important featuresare in images with psychophysics testing how those features are used to classify objects.
Revised: In our work, we complement these descriptions of where important featuresare in images with psychophysics testing how those features are used to classify objects.

Original: Comparing visual strategies of humans and machines.
Revised: Comparing visual strategies of humans and machines.

Original: As methods in explainable artificialintelligence have developed over the past decade, they have opened up opportunities for comparingthe regions selected by humans and DNNs when solving tasks.
Revised: As methods in explainable artificialintelligence have developed over the past decade, they have opened up opportunities for comparingthe regions selected by humans and DNNs when solving tasks.

Original: Many of these comparisons havefocused human image saliency measurements captured by eye tracking or mouse clicks during passiveor active viewing [20,22,37–40].
Revised: Many of these comparisons havefocused human image saliency measurements captured by eye tracking or mouse clicks during passiveor active viewing [20,22,39–42].

Original: Others have compared categorical representation distances [38,41],or combined those distances with measures of human attention [28].
Revised: Others have compared categorical representation distances [40,43],or combined those distances with measures of human attention [28].

Original: The most direct comparisonbetween human and DNN visual strategies involved analyzing the minimal image patch needed torecognize an object [9, 42].
Revised: The most direct comparisonbetween human and DNN visual strategies involved analyzing the minimal image patch neededto recognize an object [9, 44, 45].

Original: However, such studies were limited, comparing humans with severalolder DNNs using only a handful of images.
Revised: However, such studies were limited, comparing humans withseveral older DNNs using only a handful of images.

Original: To the best of our knowledge, the largest-scalecomparison between human and DNN visual strategies involved the ClickMe dataset, which wasused to compare visual regions preferred by humans for object classification to attention models [20].
Revised: To the best of our knowledge, the largest-scalecomparison between human and DNN visual strategies involved the ClickMe dataset, which wasused to compare visual regions preferred by humans for object classification to attention models [20].

Original: What is noticeably missing from each of these studies is a large-scale study of how human and DNNalignment has changed as a function of performance.
Revised: What is noticeably missing from each of these studies is a large-scale analysis of how human and DNN alignment has changed as a function of performance.

Original: Improving the correspondence between humans and machines.
Revised: Improving the correspondence between humans and machines.

Original: Inconsistencies between humansand DNN representations can be resolved by directly training models to act more like humans.
Revised: Inconsistencies between humansand DNN representations can be resolved by directly training models to act more like humans.

Original: DNNshave been trained to have more human-like attention, or human-like representational distances in theiroutput layer [20,38,41,43].
Revised: DNNshave been trained to have more human-like attention, or human-like representational distances in theiroutput layer [20,40,43,46,47].

Original: Here, we add to these successes with the neural harmonizer, a trainingroutine designed to align the visual strategies (Fig. 1) between any two observers by minimizing thedissimilarity in their visual explanations.
Revised: Here, we add to these successes with the neural harmonizer, a trainingroutine designed to align the visual strategies (Fig. 1) between any two observers by minimizing thedissimilarity in their visual explanations.

Original:  Methods
Revised:  Methods

Original: Human feature importance datasets.
Revised: Human feature importance datasets.

Original: We focus on the ImageNet dataset to compare the visualstrategies of humans and DNNs for object recognition at scale.
Revised: We focus on the ImageNet dataset to compare the visualstrategies of humans and DNNs for object recognition at scale.

Original: We rely on the two significantefforts for gathering feature importance data from humans on ImageNet: the Clicktionary [22] and
Revised: We rely on the two significantefforts for gathering feature importance data from humans on ImageNet: the Clicktionary [22] and

Original: ClickMe [20] games, which use slightly different methods to collect their data.
Revised: ClickMe [20] games, which use slightly different methods to collect their data.

Original: Both games beginwith the same basic setup: two observers play together to locate features in an object image that theybelieve are important for classifying it.
Revised: Both games beginwith the same basic setup: two observers play together to locate features in an object image that theybelieve are important for classifying it.

Original: As one of the observers selects important features, thosefeatures are filled into a blank canvas for the other observer to classify as quickly as possible.
Revised: As one of the observers selects important features, thosefeatures are filled into a blank canvas for the other observer to classify as quickly as possible.

Original: In
Revised: In

Original: Clicktionary [22], both players are humans, whereas in ClickMe [20], the player selecting features isa human and the player recognizing images is a CNN.
Revised: Clicktionary [22], both players are humans, whereas in ClickMe [20], the player selecting features isa human and the player recognizing images is a CNN.

Original: In total, Clicktionary [22] contains featureimportance maps for 200 images from the ImageNet validation set, whereas ClickMe [20] containsfeature importance maps for 196,499 images from ImageNet training and validation sets.
Revised: In total, Clicktionary [22] captured importantfeatures for 200 images from the ImageNet validation set, whereas ClickMe [20] captured importantfeatures for 196,499 images from ImageNet training and validation sets.

Original: Thus, ClickMe has far more data than Clicktionary , but Clicktionary data has more reliable human featureimportance data thanClickMe .
Revised: Thus, ClickMe has far moredata than Clicktionary , but Clicktionary data has more reliable human feature importance data than

Revised: ClickMe (although the two were reported to be strongly correlated).

Revised: To use the visual strategiescaptured in Clicktionary and ClickMe in our experiments, we computed maps for every image thatcaptured the probability of importance at every pixel position.

Original: By comparing human and DNN alignment on  both of these datasets,we establish the robustness of our findings.
Revised: By comparing human and DNNalignment on maps from both of these datasets, we establish the robustness of our findings.

Original: Our experiments comparing human and DNN alignmentcompare feature importance maps from these datasets for the ImageNet validation set.
Revised: Ourexperiments comparing human and DNN alignment compare feature importance maps from thesedatasets for the ImageNet validation set.

Original: As we describein §4, we incorporate ClickMe feature importance maps for the ImageNet training set into our neuralharmonizer routine to align human and DNN visual strategies.
Revised: As we describe in §4, we incorporate ClickMe featureimportance maps for the ImageNet training set into our neural harmonizer routine to align human and DNN visual strategies.

Original: Psychophysics participants and dataset.
Revised: Psychophysics participants and dataset.

Original: We complemented the feature importance maps from
Revised: We complemented the feature importance maps from

Original: Clicktionary and ClickMe with psychophysics experiments on rapid visual categorization.
Revised: Clicktionary and ClickMe with psychophysics experiments on rapid visual categorization.

Original: Werecruited 199 participants from Amazon Mechanical Turk ( mturk.com ) to complete the experiments.
Revised: Werecruited 199 participants from Amazon Mechanical Turk ( mturk.com ) to complete the experiments.

Original: Participants viewed a psychophysics dataset consisted of the 100 animal and 100 non-animal imagesin the Clicktionary game taken from the ImageNet validation set [22].
Revised: Participants viewed a psychophysics dataset consisting of the 100 animal and 100 non-animal images in the Clicktionary game taken from the ImageNet validation set [22].

Original: We used the feature importancemaps for each image as masks for the object images, allowing us to control the proportion of importantfeatures observers were shown when asked to recognize objects (Fig. 5a).
Revised: We used the feature importancemaps for each image as masks for the object images, allowing us to control the proportion of importantfeatures observers were shown when asked to recognize objects (Fig. 5a).

Original: We generated versionsof each image that revealed anywhere between 1% to 100% (at log-scale spaced intervals) of the B u c k e t M o b il e B o w t i e
Revised: We generated versionsof each image that revealed anywhere between 1% to 100% (at log-scale spaced intervals) of theimportant object pixels (see SI §1 for details on mask generation).

Original: B a ll

Original: B a n j o

Original: ResNet

Original: [Figure]

Original: ResNet

Original: MobileNet ResNet 50 ConViTSimCLR MobileNet ConvNext important object pixels (see SI §1 for details on mask generation). The total number of pixels revealedin each image was was equal for every image at a given level of image masking.
Revised: The total number of pixels revealedwas equal for every image at a given level of image masking.

Original: The revealedimagefeatures were centered in each image  to maximize the likelihood they fell on a partcipant’s fovea.
Revised: The revealed image features werecentered to maximize the likelihood they fell on a partcipant’s fovea.

Original: Each participant saw only one masked version of each object image.
Revised: Each participant saw only onemasked version of each object image.

Original: Psychophysics experiment.
Revised: Psychophysics experiment.

Original: Participants were instructed to categorize images in the psychophysicsdataset as animals or non-animals as quickly and accurately as possible.
Revised: Participants were instructed to categorize images in the psychophysicsdataset as animals or non-animals as quickly and accurately as possible.

Original: Each experimental trialconsisted of the following sequence of events overlaid onto a white background (SI Fig.
Revised: Each experimental trialconsisted of the following sequence of events overlaid onto a white background (SI Fig.

Original: 1): ( i )a fixation cross displayed for a variable time (1,100–1,600ms); ( ii ) an image for 400ms; ( iii ) anadditional 150ms of response time.
Revised: 1): ( i )a fixation cross displayed for a variable time (1,100–1,600ms); ( ii ) an image for 400ms; ( iii ) anadditional 150ms of response time.

Original: In other words, the experiment forced participants to performrapid object classification.
Revised: In other words, the experiment forced participants to performrapid object classification.

Original: They were given a total of 550ms to view an image and press a button toindicate its category (feedback was provided on trials in which responses were not provided withinthis time limit).
Revised: They were given a total of 550ms to view an image and press a button toindicate its category (feedback was provided on trials in which responses were not provided withinthis time limit).

Original: Images were sized at 256 x 256 pixel resolution, which is equivalent to a stimulussize between approximately 5 – 11 degrees of visual angle across a likely range of possible displayand seating setups participants used for the experiment.
Revised: Images were sized at 256 x 256 pixel resolution, which is equivalent to a stimulussize between approximately 5 – 11 degrees of visual angle across a likely range of possible displayand seating setups participants used for the experiment.

Original: Similar paradigms and timing parametershave been shown to capture pre-attentive visual system processing [31,44–46].
Revised: Similar paradigms and timing parametershave been shown to capture pre-attentive visual system processing [31,48–50].

Original: Participants providedinformed consent electronically and were compensated $3.00 for their time ( ∼ 10–15 min; $15.00/hr).
Revised: Participants providedinformed consent electronically and were compensated $3.00 for their time ( ∼ 10–15 min; $15.00/hr).

Original: Models.
Revised: Models.

Original: We compared humans with 85 different DNNs representing the variety of approachesused in the field today: 58 CNNs trained on ImageNet [1,47–59,59–68], 4 CNNs trained on otherdatasets in addition to ImageNet (which we  to as “CNN extra data”) [1, 61, 69] 4 visiontransformers [70– CNNs supervision [
Revised: We compared humans with 85 different DNNs representing the variety of approachesused in the field today: 58 CNNs trained on ImageNet [1,51–63,63–72], 4 CNNs trained on otherdatasets in addition to ImageNet (which we refer to as “CNN extra data”) [1, 65, 73], 4 visiontransformers [74–78], 6 CNNs trained with self-supervision [79, 80], and 13 models trained forrobustness to noise or adversarial examples [81,82].

Original: robustness
Revised: We used pretrained weights for each of thesemodels supplied by their authors, with a variety of licenses (detailed in SI §2), implemented in Tensorflow 2.0, Keras, or PyTorch.

Original:  Results
Revised:  Results

Original: 4.1 Where are diagnostic object features for humans and DNNs?
Revised: 4.1 Where are diagnostic object features for humans and DNNs?

Original: To systematically compare the visual strategies of object recognition for humans and DNNs on
Revised: To systematically compare the visual strategies of object recognition for humans and DNNs on

Original: ImageNet, we first turned to the ClickMe dataset of feature importance maps [20].
Revised: ImageNet, we first turned to the ClickMe dataset of feature importance maps [20].

Original: In order to derivecomparable feature importance maps for DNNs, we needed a method that could be efficiently andconsistently applied to each of the 85 DNNs we tested with out any idiosyncratic hyperparameters.
Revised: In order to derivecomparable feature importance maps for DNNs, we needed a method that could be efficiently and consistently applied to each of the 85 DNNs we tested without any idiosyncratic hyperparameters.

Original: This led us to choose a classic method for explainable artificial intelligence, image featuresaliency [79].
Revised: Thisled us to choose a classic method for explainable artificial intelligence, image feature saliency [83].

Original: We prepared human feature importance maps from ClickMe by taking the averageimportance map across humans in every image that also appeared in ImageNet validation.
Revised: We prepared human feature importance maps from ClickMe by taking the average importance mapproduced by humans for every image that also appeared in ImageNet validation.

Original: We thenrecorded each model’s top-1 accuracy on the entire ImageNet validation set and used Spearman’srank-correlation to measure the similarity between  feature mapsof humans and DNNs for each image.
Revised: We then used Spearman’s rank-correlation to measure the similarity between human feature maps and DNN featuremaps for each image [48].

Original: We also computed the maximum possible alignment for human feature importance maps by recordingthe correlation in every image between two randomly selected groups of participants.
Revised: We also computed the inter-rater alignment of human feature importancemaps as the mean split-half correlation across 1000 random splits of the participant pool.

Original: We storedthe average correlation after repeating this procedure 1000 times, and then normalized each DNN’scorrelation with humans by this score.
Revised: We thennormalized each human-DNN correlation by this score, making 100% alignment possible [20].

Original: There were dramatic qualitative differences between the features selected by humans and DNNson ImageNet.
Revised: There were dramatic qualitative differences between the features selected by humans and DNNs on ImageNet.

Original: In general, humans selected less contextual information andfocused more on objectdetails: the handle and opening of a bucket, the chassis of a snow mobile, or the frets of a banjo(Fig. 2).
Revised: In general, humans selected less context and more object detail: the handle and openingof a bucket, the chassis of a snow mobile, or the frets of a banjo (Fig. 2).

Original: The DNN that was most aligned with humans, the DenseNet121, was still only 58% alignedwith humans (Fig. 3).
Revised: The DNN that was mostaligned with humans, the DenseNet121, was still only 58% aligned with humans (Fig. 3).

Original: Plotting the relationship between DNN ImageNet accuracy and alignment with humans revealed astriking trade-off: as model performance has improved beyondthe DenseNet121, their alignmentwith humans has worsened (Fig. 3).
Revised: Plotting the relationship between DNNs’ top-1 accuracy on ImageNet with their human alignmentrevealed a striking trade-off: as the accuracy of DNNs has improved beyond DenseNet121, theiralignment with humans has worsened (Fig. 3).

Original: For example, consider the ConvNext [1], which achieved thebest top-1 accuracy in our experiments (85.8%), was only 22% aligned with humans – equivalent tothe alignment of the BagNet33 [64] (63% top-1 accuracy).
Revised: For example, consider the ConvNext [1], whichachieved the best top-1 accuracy in our experiments (85.8%), was only 22% aligned with humans –equivalent to the alignment of the BagNet33 [68] (63% top-1 accuracy).

Revised: As an additional control, wecomputed the similarity between the average ClickMe map, which exhibits a center bias [84,85] (SI

Revised: Fig. 5), and each individual ClickMe map.

Original: {+This center-bias control was only outperformed by 21/
+}
Revised: {+This center-bias control was only outperformed by 21/
+}

Revised: CNNs we tested ( † in Fig. 3).

Original: Overall, we observe that human and DNNalignment has considerably worsened since the introduction of these two models. 
Revised: Overall, we observe that human and DNN alignment has considerablyworsened since the introduction of these two models. B u c k e t M o b il e B o w t i e

Original: The neural harmonizer.
Revised: B a ll B a n j o ResNet50 (Neural harmonizer)

Revised: Human ( ClickMe )

Revised: ResNet

Revised: MobileNet

Original:  While scaling DNNs has had an immense impact on improving theirperformance on popular benchmark tasks, little has changed regarding the fundamental differences inarchitectures between DNNs and the human visual system [35].
Revised: ResNet 50 ConViTSimCLR MobileNet ConvNext The neural harmonizer. While scaling DNNs has had an immense impact on improving theirperformance on popular benchmark tasks, little has changed regarding the fundamental differences inarchitectures between DNNs and the human visual system [37].

Original: While it’s plausible that introducingmore biological constraints into DNNs could improve their alignments with humans, there’s evidencethat doing so would hurt benchmark performance and require bespoke development for every differentarchitecture [80–82].
Revised: While it is plausible that introducingmore biological constraints into DNNs could improve their alignments with humans, there is evidencethat doing so would hurt benchmark performance and require bespoke development for every differentarchitecture [86–88].

Original: Is there a way to align any DNN’s visual strategies with humans without hurtingits performance?
Revised: Is it possible to align a DNN’s visual strategies with humans without hurting itsperformance?

Original: Such a general-purpose method for aligning human and DNN visual strategies should satisfy thefollowing criteria: ( i ) Models trained with the method  have feature importance maps that are alignedwith humans across feature scales.While humans are scale-invariant [33,83], DNNs typically arenot; this constraint means that alignment must hold across scales.
Revised: Such a general-purpose method for aligning human and DNN visual strategies should satisfy thefollowing criteria: ( i ) While humans are scale-invariant [33, 89], DNNs typically are not. Thus,models trained with the method need to have feature importance maps that are aligned with humansacross feature scales.

Original: ( ii )
Revised: ( ii )

Original: The method should work withany fully-differentiable network architecture.
Revised: The method should work with any fully-differentiable network architecture.

Original: ( iii ) Task-accuracy of a model trained with the methodshould not be worse than a model trained without the method.
Revised: ( iii ) Task-accuracy of a model trained with the method should not be worse than a model trainedwithout the method.

Original: We created the neural harmonizer tosatisfy each of these criteria.
Revised: We created the neural harmonizer to satisfy each of these criteria.

Original: Let us consider a supervised classification problem with an input space X an output space Y ⊆
Revised: Let us consider a supervised classification problem with an input space, X an output space Y ⊆

Original: R cand a predictor function f θ : X → Y parameterized by θ , which maps an input vector x ∈ X toan output f θ ( x ) .
Revised: R cand a predictor function f θ : X → Y parameterized by θ , which maps an input vector x ∈ X toan output f θ ( x ) .

Original: We denote g : F × X → X an explanation functional that, given a predictorf θ ∈ F and an input, returns a feature importance map ϕ =
Revised: We denote g : F × X → X an explanation functional that, given a predictorf θ ∈ F and an input, returns a feature importance map ϕ =

Original: g ( f θ , x ) .
Revised: g ( f θ , x ) .

Original: Here, we focus on DNNsaliency as our method for computing feature importance in DNNs, but the method can work withany differentiable network explanation method (see SI §2 for details).
Revised: Here, we focus on DNNsaliency as our method for computing feature importance in DNNs, but the method can work withany differentiable network explanation method.

Original: To satisfy criterion ( i ), the neural harmonizer will enforce alignment across feature importancemap scales.
Revised: To satisfy criterion ( i ), the neural harmonizer will enforce alignment across feature importancemap scales.

Original: Let P i ( . ) be the function mapping a feature importance map ϕ to it’s representationin the N levels of a Gaussian pyramid, with i ∈ { 1 , ..., N } .
Revised: Let P i ( . ) be the function mapping a feature importance map ϕ to it is representationin the N levels of a Gaussian pyramid, with i ∈ { 1 , ..., N } .

Original: The function P i ( ϕ ) is computed bydownsampling P
Revised: The function P i ( ϕ ) is computed bydownsampling P

Original: i − 1 ( ϕ ) using a Gaussian kernel, with P 1 ( ϕ ) =
Revised: i − 1 ( ϕ ) using a Gaussian kernel, with P 1 ( ϕ ) =

Original: ϕ .
Revised: ϕ .

Original: We then seek to minimize 217 (cid:80)
Revised: We then seek to minimize 218 (cid:80)

Original: Ni ||P
Revised: Ni ||P

Original: i
Revised: i

Original: ( g ( f θ , x ))
Revised: ( g ( f θ , x ))

Original: − P i ( ϕ ) || 2 , which will align feature importance maps between humans and
Revised: − P i ( ϕ ) || 2 , which will align feature importance maps between humans and

Original: DNNs at every scale of the pyramid.
Revised: DNNs at every scale of the pyramid.

Original: Furthermore, because this approach is fully differentiable, italso partially satisfies criterion ( ii ).
Revised: Furthermore, because this approach is fully differentiable, italso partially satisfies criterion ( ii ).

Original: [Figure]
Revised: [Figure]

Original: To completely satisfy criterion ( ii ), the neural harmonizer should work well with training routinesdesigned for large-scale computer vision challenges like ImageNet.
Revised: To completely satisfy criterion ( ii ), the neural harmonizer should work well with training routinesdesigned for large-scale computer vision challenges like ImageNet.

Original: This means the neural harmonizermust avoid optimization issues at scale.
Revised: This means the neural harmonizermust avoid optimization issues at scale.

Original: To do this, we need a way of comparing feature importancemaps between humans and DNNs that is invariant to the norm of either map.
Revised: To do this, we need a way of comparing feature importancemaps between humans and DNNs that is invariant to the norm of either map.

Original: We therefore standardize feature importance maps from humans and DNNs before comparing them, and only measurealignment on the most important areas of the image for each observer.
Revised: We therefore standardize feature importance maps from humans and DNNs before comparing them, and only measurealignment on the most important areas of the image for each observer.

Original: Formally, let z : X → X be astandardization function over feature importance maps that takes the mean and standard deviationcomputed of each map ϕ such that z ( ϕ ) has 0 activation on average and unit standard deviation.
Revised: Formally, let z : X → X be astandardization function over feature importance maps that takes the mean and standard deviationcomputed for each map ϕ such that z ( ϕ ) has 0 activation on average and unit standard deviation.

Original: To focus alignment on important regions, let z ( ϕ ) + denote the positive part of the standardizedexplanation z ( ϕ ) .
Revised: To focus alignment on important regions, let z ( ϕ ) + denote the positive part of the standardizedexplanation z ( ϕ ) .

Original: Finally, to satisfy criterion ( iii ) and train models that are at least as accurate asthose trained without the neural harmonizer, we include the familiar cross entropy objective to yieldthe complete neural harmonizer loss: 
Revised: Finally, to satisfy criterion ( iii ) and train models that are at least as accurate asthose trained without the neural harmonizer, we include the familiar cross entropy objective to yieldthe complete neural harmonizer loss: [Equation] [Equation] +

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: Training.
Revised: Training.

Original: We trained two different DNNs with the neural harmonizer, ResNet50 and VGG16, whichwere selected because they are popular networks that sit on the boundary of the trade-off between
Revised: We trained four different DNNs with the neural harmonizer, ResNet50, VGG16, EfficientNet_b0, and MobileNet V1, each of which sit on the boundary of the trade-off between

Original: DNN performance and alignment with humans.
Revised: DNN performance and alignment with humans.

Original: Models were trained using the neural harmonizerto optimize classification performance on ImageNet and feature importance map alignment withhuman data from ClickMe .
Revised: Models were trained using the neural harmonizerto optimize classification performance on ImageNet and feature importance map alignment withhuman data from ClickMe .

Original: We trained models on all images in the ImageNet training set, but because
Revised: We trained models on all images in the ImageNet training set, but because

Original: ClickMe only contains human feature importance maps for a portion of those images, we computedthe classification loss but not the alignment loss in Eq.
Revised: ClickMe only contains human feature importance maps for a portion of those images, we computedthe classification loss but not the alignment loss in Eq.

Original: (1) for images without importance maps.
Revised: (1) for images without importance maps.

Original: Models were trained using 8 cores V4 TPUs on the google cloud platform, and training lastedapproximately one day.
Revised: Models were trained using 8 cores V4 TPUs on the Google Cloud Platform, and training lastedapproximately one day.

Original: Models were trained with an augmented ResNet training recipe (built fromhttps://github.com/tensorflow/tpu/ ).
Revised: Models were trained with an augmented ResNet training recipe (built fromhttps://github.com/tensorflow/tpu/ ).

Original: Models were optimized with SGD and momentumover batches of 512 images, a learning rate of 0 . 3 , and label smoothing [84].
Revised: Models were optimized with SGD and momentumover batches of 512 images, a learning rate of 0 . 3 , and label smoothing [90].

Original: Images were augmentedwith random left-right flips, mixup [85].
Revised: Images were augmentedwith random left-right flips and mixup [91].

Original: The learning rate was adjusted over the course of trainingwith a schedule that began with an initial warm-up period of 5 epochs and then decaying according to a cosine function over 90 epochs, with decay at step 30, 50 and 80.
Revised: The learning rate was adjusted over the course of training with a schedule that began with an initial warm-up period of 5 epochs and then decaying accordingto a cosine function over 90 epochs, with decay at step 30, 50 and 80.

Original: The number of steps perepoch was multiplied by a factor of 3.
Revised: The number of steps perepoch was multiplied by a factor of 3.

Original: We validated that a ResNet50 and VGG16 trained withthese hyperparameters and schedule using standard cross-entropy (instead of the neural harmonizer)matched published performance.
Revised: We validated that a ResNet50 and VGG16 trained withthese hyperparameters and schedule using standard cross-entropy (instead of the neural harmonizer)matched published performance.

Original: The neural harmonizer aligns human and DNN visual strategies.
Revised: The neural harmonizer aligns human and DNN visual strategies.

Original: Returning to the observedtrade-off between ImageNet accuracy and model alignment with ClickMe human feature importancemaps, we found that training models with the neural harmonizer broke this trade-off.
Revised: Returning to the observedtrade-off between ImageNet accuracy and model alignment with ClickMe human feature importancemaps, we found that training models with the neural harmonizer broke this trade-off.

Original: All three ofour neural harmonizer models were significantly more aligned with feature importance maps andperformed better on ImageNet (Fig. 3).
Revised: All of our neuralharmonizer models were significantly more aligned with feature importance maps and performedbetter on ImageNet (Fig. 3).

Original: The changes in feature importance are dramatic: a ResNet50trained with the neural harmonizer had feature importance maps that are far less reliant on context(Fig.
Revised: The changes in feature importance are dramatic: a ResNet50 trainedwith the neural harmonizer had feature importance maps that are far less reliant on context (Fig.

Original: 1) and 20% more aligned with humans (Fig. 3; 69.17% alignment) than the standard ResNet50.
Revised: 1)and 20% more aligned with humans (Fig. 3; 69.17% alignment) than the standard ResNet50.

Original: The same model also performed 2% better in top-1 accuracy, matching the performance of a DenseNet121, without any changes to its architecture.
Revised: Thesame model also performed 2% better in top-1 accuracy, matching the performance of a DenseNet121, without any changes to its architecture.

Original: A VGG16 trained with the neural harmonizer had featureimportance maps that were 269% more aligned with humans (94.45% alignment) and performed 3%better in top-1 accuracy than a standard VGG16.
Revised: A VGG16 trained with the neural harmonizer had featureimportance maps that were 269% more aligned with humans (94.45% alignment) and performed 3%better in top-1 accuracy than a standard VGG16.

Original: [Figure]
Revised: [Figure]

Original: Clicktionary.
Revised: Clicktionary.

Original: To test if the trade-off between DNN ImageNet accuracy and alignment with humansis  general phenomenon we next turned to Clicktionary [22].
Revised: To test if the trade-off between DNN ImageNet accuracy and alignment with humansis a general phenomenon we next turned to Clicktionary [22].

Original: Indeed, after comparing DNNs andhumans using the same approach we used for ClickMe , we observed a similar trade-off: as DNNaccuracy has improved on ImageNet, feature importance alignment with participants in Clicktionaryhas worsened (Fig. 4).
Revised: Indeed, after testing our set of DNNson Clicktionary we observed a similar trade-off as we found for ClickMe : alignment with humanfeature importance from Clicktionary has worsened as DNN accuracy has improved on ImageNet(Fig. 4).

Original: As with ClickMe , DNNs trained with the neural harmonizer shift the accuacyalignment trade-off on this dataset.
Revised: As with ClickMe , DNNs trained with the neural harmonizer shift the accuacy-alignmenttrade-off on this dataset.

Original: 4.2 How do humans and DNNs integrate diagnostic object features into decisions?
Revised: 4.2 How do humans and DNNs integrate diagnostic object features into decisions?

Original: The trade-off we discovered between DNN accuracy on ImageNet and alignment with human visualfeature importance suggests that the two use different visual strategies.
Revised: The trade-off we discovered between DNN accuracy on ImageNet and alignment with human visualfeature importance suggests that the two use different visual strategies.

Original: However, there is potentialfor an even deeper problem.
Revised: However, there is potential for an even deeper problem.

Original: Even if two observers deem the same regions of an image as importantfor recognizing it, there is no guarantee that they use the selected features in the same way to render Animal Non-animal  100 I m p o r t a n t hu m
Revised: Even if two observers deem the same regions of an image as importantfor recognizing it, there is no guarantee that they use the selected features in the same way to rendertheir decisions.

Original: a n f e a t u r e s ( % )

Original: O r

Original: i g i n a l

Original: Human features revealed (%)

Original: N o r m a li z e d p e r f o r m a n c e ( % )

Original: Top-1 ImageNet Accuracy (%)

Original: [Figure]

Original: - l o g ( m e a n s q u a r e e rr o r )

Original: [Figure]

Original: CNNTransformerSelf -supervised

Original: RobustCNN extra data

Original: Neural harmonizer

Original: Human [Figure] their decisions. We posit that if two observers have aligned visual strategies, the will agree on bothwhere important features are in an image and how they use those features for decisions.
Revised: We posit that if two observers have aligned visual strategies, the will agree on bothwhere important features are in an image and how they use those features for decisions.

Original: We developed a psychophysics experiment to measure how different humans use features in ImageNetimages to recognize objects.
Revised: We developed a psychophysics experiment to measure how different humans use features in ImageNetimages to recognize objects.

Original: Participants viewed versions of these images where only a proportion ofthe features that were deemed most important in the Clicktionary game were visible (Fig. 5a), and had to accurately detect whether or not the image contained an animal within 550ms.
Revised: Participants viewed versions of these images where only a proportionof the features that were deemed most important in the Clicktionary game were visible (Fig. 5a). Participants had to accurately detect whether or not the image contained an animal within 550ms,which forced them to rely on feedforward processing as much as possible [33].

Original: This rapidcategorization paradigm controlled for confounds in our comparisons with the feedfoward DNNs wetested by minimizing the likelihood that participants relied on anything but feedforward processing. Participant performance on this experiment revealed how their feedforward processes integratedfeatures into accurate decisions.
Revised: Each of the 200images we used were shown to a single participant only once, meaning that we constructed decisioncurves across participants, which showed how accurately the average human converted any givenproportion of important features into an object decision.

Original: We performed the same experiment on DNNs as we did on humans,recording animal vs. non-animal decisions according to whether or not the most probable categoryin the model’s 1000-category output was an animal.
Revised: We performed the same experiment on DNNs as we did on humans, recording animal vs. non-animal decisions according to whether ornot the most probable category in the model’s 1000-category output was an animal.

Original: Because the experiment was speeded, humanparticipants did not achieve perfect accuracy.
Revised: Because theexperiment was speeded, humans did not achieve perfect accuracy.

Original: Thus, we normalize performance for humans and DNNsin order to compare the rate at which each integrates featuers into visual decisions.
Revised: Thus, we normalized performancefor humans and DNNs to compare the rate at which each integrated features into accurate decisions.

Original: We discovered a similar trade-off between ImageNet accuracy and alignment with human visualdecision making in this experiment as we did in ClickMe and Clicktionary (Fig. 5b).
Revised: We discovered a similar trade-off between ImageNet accuracy and alignment with human visualdecision making in this experiment as we did in ClickMe and Clicktionary (Fig. 5b).

Original: Indeed, themodel that was most aligned with human decision-making – the BagNet33 [64] – only achieved63.0% accuracy on ImageNet.
Revised: Indeed, themodel that was most aligned with human decision-making – the BagNet33 [68] – only achieved63.0% accuracy on ImageNet.

Original: Surprisingly,  modelstrained with the neural harmonizer broke thistrend, particularly the harmonized ResNet50 (Fig.
Revised: Surprisingly, harmonized models broke this trend, particularly theharmonized ResNet50 (Fig.

Original: 5b, top-right), despite no explicit constraints inthat procedure which forced consistent decision-making with humans.
Revised: 5b, top-right), despite no explicit constraints in that procedure whichforced consistent decision-making with humans.

Original: In contrast, a performant DNNlike ViT (85.8% accuracy) integrates information more slowly than humans or  modelstrained withthe neural harmonizer (Fig. 5c).
Revised: In contrast, a performant DNN like ViT (85.8%accuracy) integrates information slower than humans or harmonized models (Fig. 5c).

Original:  Conclusion
Revised:  Conclusion

Original: Models that reliably categorize objects like humans do would shift the paradigms of the cognitivesciences and artificial intelligence.
Revised: Models that reliably categorize objects like humans do would shift the paradigms of the cognitivesciences and artificial intelligence.

Original: But despite continuous progress over the past decade on the
Revised: But despite continuous progress over the past decade on the

Original: ImageNet benchmark, we find that DNNs are becoming worse models of human vision.
Revised: ImageNet benchmark, we find that DNNs are becoming worse models of human vision.

Original: Our solutionto this problem, the neural harmonizer routine, can be applied to any DNN to align their visualstrategies with humans and, at least in the models we tested, even improve performance.
Revised: Our solutionto this problem, the neural harmonizer routine, can be applied to any DNN to align their visualstrategies with humans and, at least in the models we tested, even improve performance.

Original: The scaling laws that are governing DNN development today are choosing to forgo human alignmentin exchange for incredible performances on nearly every benchmark in artificial intelligence.
Revised: The scaling laws that are governing DNN development today are forgoing human alignment inexchange for incredible performances on nearly every benchmark in artificial intelligence.

Original: Weencourage the field to re-analyze the costs and benefits of this exchange, particularly in light of thegrowing worries about DNNs leveraging shortcuts and dataset biases to achieve high performance [19].
Revised: We encourage the field to re-analyze the costs and benefits of this exchange, particularly in light of the growingconcerns about DNNs leveraging shortcuts and dataset biases to achieve high performance [19].

Original: Alignment with human vision need not be exchanged with performance if DNNs are trained withour neural harmonizer.
Revised: Alignment with human vision need not be exchanged with performance if DNNs are trained with ourneural harmonizer.

Original: Our project code ( https://tinyurl.com/metapred ) can be used toincorporate the neural harmonizer in any new DNN created and measure its alignment with humanson the datasets we describe in this paper.
Revised: Our codebase ( https://tinyurl.com/metapred ) can be used to incorporatethe neural harmonizer in any new DNN created and measure its alignment with humans on thedatasets we describe in this paper.

Original: Limitations.
Revised: Limitations.

Original: One possible explanation for the misalignment between DNNs and humans thatwe observe is that recent DNNs have achieved superhuman accuracy on ImageNet.
Revised: One possible explanation for the misalignment between DNNs and humans thatwe observe is that recent DNNs have achieved superhuman accuracy on ImageNet.

Original: Superhuman
Revised: Superhuman

Original: DNNs have been described in biomedical applications [87,88] where there’s definitive biologicalground-truth labels, but ImageNet labels are noisy, making it unclear if such an achievement islaudable ( https://labelerrors.com/ ).
Revised: DNNs have been described in biomedical applications [93,94] where there is definitive biologicalground-truth labels, but ImageNet labels are noisy, making it unclear if such an achievement islaudable ( https://labelerrors.com/ ).

Original: Thus, an equally likely explanation is that the continuedimprovements of DNNs at least partially reflect their exploitation of shortcuts in ImageNet [19].
Revised: Thus, an equally likely explanation is that the continuedimprovements of DNNs at least partially reflect their exploitation of shortcuts in ImageNet [19].

Revised: The scope of our work is also limited in that it focuses on object recognition in ImageNet.

Original: Weleave experiments that can adjudicate between these possibilities to future work, and recommendpractitioners use the neural harmonizer to ensure trustworthy behavior in DNNs.
Revised: It ispossible that models trained on other tasks, such as segmentation, may be more aligned with humans. Finally, our modeling efforts were hamstrung for the largest-scale models in existence. Our workdoes not answer how much harmonization would help models like Visual Transformers or CLIPbecause of the massive investment needed to train them. The Neural Harmonizer method will scale tothese models but it is possible that more ClickMe human feature importance maps are needed for thehundreds-of-millions or billions of images used to harmonize successfully.

Original: Broader impacts.
Revised: Broader impacts.

Original: A persistent issue in the field of artificial intelligence is the tendency of modelsto exploit dataset biases.
Revised: A persistent issue in the field of artificial intelligence is the tendency of models toexploit dataset biases.

Revised: A central theme of our work is that there are facets of human perception that are

Revised: Animal

Original: With a growing trend towards automating every facet of society with DNNs,models exploiting such biases have the potential to learn strategies that systematically oppress groupsof people.
Revised: Non-animal  100 I m p o r t a n t hu m a n f e a t u r e s ( % ) O r

Original: A central theme of our work is that there’s elements of human vision that are not capturedby DNNs, particularly those which follow the scaling laws which have been so embraced by industryleaders.
Revised: i g i n a l Human features revealed (%) N o r m a li z e d p e r f o r m a n c e ( % ) Top-

Revised: [Figure]

Revised: - l o g ( m e a n s q u a r e e rr o r )

Revised: [Figure]

Revised: CNNTransformerSelf -supervised

Revised: RobustCNN extra data

Original:  Forcing DNNs to rely on similar visual strategies as humans could represent a scalable pathforward to correcting the insidious biases which have assailed applications of artificial intelligence.At the very least, this strategy would make DNNs more predictable and interpretable for their humancreators, to make it easier to filter out bad models.
Revised: Neural harmonizer Human    not captured by DNNs, particularly those which follow the scaling laws which have been so embracedby industry leaders. Forcing DNNs to rely on similar visual strategies as humans could represent a scalable path forward to correcting the insidious biases which have assailed under-constrained modelsof artificial intelligence.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Methods.
Revised: [Yes] See Methods.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Discussion.(c)
Revised: [Yes] See Discussion.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See
Revised: [Yes] See

Original: Discussion
Revised: Discussion

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi- ments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] See Methods
Revised: [Yes] See Methods

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See Methods
Revised: [Yes] See Methods

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [Yes] See SI.
Revised: [Yes] See SI.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [Yes] See Methods
Revised: [Yes] See Methods