Original: Simulating Human Gaze with Neural Visual Attention
Revised: Simulating Human Gaze with Neural Visual Attention

Original: Abstract
Revised: Abstract

Original: Existing models of human visual attention are generally unable to incorporate direct task guidance and therefore cannot model an intent or goal when exploring a scene.
Revised: Existing models of human visual attention are generally unable to incorporate direct task guidance and therefore cannot model an intent or goal when exploring a scene.

Original: To integrate guidance of any downstream visual task into attention modeling, we propose the Neural Visual Attention (NeVA) algorithm.
Revised: To integrate guidance of any downstream visual task into attention modeling, we propose the Neural Visual Attention (NeVA) algorithm.

Original: To this end, we impose to neural networks the biological constraint of foveated vision and train an attention mechanism to generate visual explorations that maximize the performance with respect to the downstream task.
Revised: To this end, we impose to neural networks the biological constraint of foveated vision and train an attention mechanism to generate visual explorations that maximize the performance with respect to the downstream task.

Original: We observe that biologically constrained neural networks generate human-like scanpaths without being trained for this objective.
Revised: We observe that biologically constrained neural networks generate human-like scanpaths without being trained for this objective.

Original: Extensive experiments on three common benchmark datasets show that our method outperforms state-of-the-art unsupervised human attention models in generating human-like scanpaths. 
Revised: Extensive experiments on three common benchmark datasets show that our method outperforms state-of-the-art unsupervised human attention models in generating human-like scanpaths. Full paper available at TMLR: https:// openreview .net/forum?id= 7iSYW1FRWA .

Original:  Introduction
Revised:  Introduction

Original: Computational modeling of human visual attention lies at the intersection of many disciplines, such as neuroscience, cognitive psychology, and computer vision.
Revised: Computational modeling of human visual attention lies at the intersection of many disciplines, such as neuroscience, cognitive psychology, and computer vision.

Original: Despite eye-tracking technologies becoming increasingly cost-efficient, collecting human expert gaze data in domain-specific applications, e.g., medical or higher education fields, remains expensive.
Revised: Despite eye-tracking technologies becoming increasingly cost-efficient, collecting human expert gaze data in domain-specific applications, e.g., medical or higher education fields, remains expensive.

Original: Therefore, models simulating plausible task-specific scanpaths are highly required both for understanding the biological mechanism [14, 21] as well as in applications [19, 10, 18, 7].
Revised: Therefore, models simulating plausible task-specific scanpaths are highly required both for understanding the biological mechanism [14, 21] as well as in applications [19, 10, 18, 7].

Original: Most eye-tracking datasets and related methods are designed for saliency prediction [2, 23].
Revised: Most eye-tracking datasets and related methods are designed for saliency prediction [2, 23].

Original: However, saliency maps are static and cannot describe the temporal aspect of visual exploration.
Revised: However, saliency maps are static and cannot describe the temporal aspect of visual exploration.

Original: On the other end, scanpath models have been proposed that can generate fixation sequences.
Revised: On the other end, scanpath models have been proposed that can generate fixation sequences.

Original: The circuitry of winner-take-all [14] can be combined with any saliency estimation method, such as Itti’s saliency map [11] to generate scanpaths in an unsupervised manner.
Revised: The circuitry of winner-take-all [14] can be combined with any saliency estimation method, such as Itti’s saliency map [11] to generate scanpaths in an unsupervised manner.

Original: Boccignone et al.
Revised: Boccignone et al.

Original: [3] generate gaze trajectories based on non-local transition probabilities defined in a saliency field.
Revised: [3] generate gaze trajectories based on non-local transition probabilities defined in a saliency field.

Original: Recently, [22] described attention by mean of gravitational laws of motion, where visual features are considered as masses attracting the focus of attention.
Revised: Recently, [22] described attention by mean of gravitational laws of motion, where visual features are considered as masses attracting the focus of attention.

Original: However, existing approaches tacitly assume perfect vision by processing the input in its full resolution all at once, and do not provide a flexible way to incorporate task guidance to the generated scanpaths.
Revised: However, existing approaches tacitly assume perfect vision by processing the input in its full resolution all at once, and do not provide a flexible way to incorporate task guidance to the generated scanpaths.

Original: We propose a Neural Visual Attention (NeVA) algorithm to generate purely task-driven gaze trajectories.
Revised: We propose a Neural Visual Attention (NeVA) algorithm to generate purely task-driven gaze trajectories.

Original: The algorithm consists of three major components.
Revised: The algorithm consists of three major components.

Original: First, a differentiable foveation layer that, given a fixation position and an input image, simulates biologically plausible foveated vision.
Revised: First, a differentiable foveation layer that, given a fixation position and an input image, simulates biologically plausible foveated vision.

Original: Second, a task model that gets passed a foveated image by the foveation layer and produces a loss signal with respect to its downstream task (e.g., classifying the original class or reconstructing the original image).
Revised: Second, a task model that gets passed a foveated image by the foveation layer and produces a loss signal with respect to its downstream task (e.g., classifying the original class or reconstructing the original image).

Original: Lastly, an attention mechanism determining the next fixation position in order to minimize the loss with respect to the task model (see Figure 1 for an illustration).
Revised: Lastly, an attention mechanism determining the next fixation position in order to minimize the loss with respect to the task model (see Figure 1 for an illustration).

Original: The resulting architectureis fully differentiable.
Revised: The resulting architecture

Original: We use neural networks to model both the attention and the task model and backpropagation to train the attention mechanism.

Original: The task model is only used to generate the loss propagation

Original: Loss propagation
Revised: Loss propagation

Original: Stimulus sampling
Revised: Stimulus sampling

Original: Attention t= π
Revised: Attention t= π

Original: Stimulus
Revised: Stimulus

Original: Perceived t= h t = τ
Revised: Perceived t= h t = τ

Original: Agent State α
Revised: Agent State α

Original: Perceived t= aggregate during training and can be discarded at inference time.
Revised: Perceived t [Figure] aggregate is fully differentiable.

Revised: We use neural networks to model both the attention and the task model and backpropagation to train the attention mechanism.

Revised: The task model is only used to generate the loss during training and can be discarded at inference time.

Original: The flexibility of the framework allows us to analyze the contribution of different tasks and hence generate task-specific gaze trajectories.
Revised: The flexibility of the framework allows us to analyze the contribution of different tasks and hence generate task-specific gaze trajectories.

Original: To validate the proposed approach, we compare it to three well-established unsupervised models of visual attention and three baseline approaches.
Revised: To validate the proposed approach, we compare it to three well-established unsupervised models of visual attention and three baseline approaches.

Original: The plausibility of the models is measured as the similarity of the generated scanpath with human eye-tracking data.
Revised: The plausibility of the models is measured as the similarity of the generated scanpath with human eye-tracking data.

Original: Results demonstrate the superiority of the proposed approach.
Revised: Results demonstrate the superiority of the proposed approach.

Original:  Neural Visual Attention (NeVA)
Revised:  Neural Visual Attention (NeVA)

Original: We propose an algorithm for Neural Visual Attention (NeVA) which enables to generate scanpaths of visual attention under the guidance of any differentiable task model.
Revised: We propose an algorithm for Neural Visual Attention (NeVA) which enables to generate scanpaths of visual attention under the guidance of any differentiable task model.

Original: As previously mentioned, this is made by three main components: a task model, a foveation layer, and an attention mechanism.
Revised: As previously mentioned, this is made by three main components: a task model, a foveation layer, and an attention mechanism.

Original: In what follows, we formally define each component of the model.
Revised: In what follows, we formally define each component of the model.

Original: The approach is illustrated in
Revised: The approach is illustrated in

Original: Figure 1.
Revised: Figure 1.

Original: Task model.
Revised: Task model.

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: { S 1 , ..., S N } be a collection of input stimuli, and Y = { y 1 , ..., y N } be a set of corresponding targets (i.e., they correspond to class labels in the case of a classification task).
Revised: { S 1 , ..., S N } be a collection of input stimuli, and Y = { y 1 , ..., y N } be a set of corresponding targets (i.e., they correspond to class labels in the case of a classification task).

Original: A task model τ is trained to map input stimuli to their corresponding targets, such that
Revised: A task model τ is trained to map input stimuli to their corresponding targets, such that

Original: [Equation]
Revised: [Equation]

Original: Foveation layer.
Revised: Foveation layer.

Original: A fully differentiable foveation layer is defined to simulate human vision, constrained by its structure to fine vision in corresponding to the fovea, and coarse vision in the periphery.
Revised: A fully differentiable foveation layer is defined to simulate human vision, constrained by its structure to fine vision in corresponding to the fovea, and coarse vision in the periphery.

Original: Given a stimulus S , the foveation mechanism computes the perceived stimulus π ( S, ξ t ) as a foveated rendering of S centered at the current focus of attention ξ t .
Revised: Given a stimulus S , the foveation mechanism computes the perceived stimulus π ( S, ξ t ) as a foveated rendering of S centered at the current focus of attention ξ t .

Original: In particular, let ˜ S be a coarse version of S , obtained by applying a convolution to the original stimulus S to suppress any high frequency components.
Revised: In particular, let ˜ S be a coarse version of S , obtained by applying a convolution to the original stimulus S to suppress any high frequency components.

Original: Then, the foveated stimulus is obtained as a linear combination of the original stimulus and its coarse version,
Revised: Then, the foveated stimulus is obtained as a linear combination of the original stimulus and its coarse version,

Original: [Equation]
Revised: [Equation]

Original: where G σ ξ ( t ) is a Gaussian blob, with mean ξ t and standard deviation σ ξ , centered at ξ t .
Revised: where G σ ξ ( t ) is a Gaussian blob, with mean ξ t and standard deviation σ ξ , centered at ξ t .

Original: To incorporate past information, we define the internal agent’s state h t ∼ h ( S, ξ t ) to express the cumulative perceived information, or memory of the system, such that
Revised: To incorporate past information, we define the internal agent’s state h t ∼ h ( S, ξ t ) to express the cumulative perceived information, or memory of the system, such that

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: where [ ... ] 0:1 is an element-wise clipping operator with minimum 0 and maximum 1 , and J d is a d × d dimensional unit matrix (matrix only filled with ones).
Revised: where [ ... ] 0:1 is an element-wise clipping operator with minimum 0 and maximum 1 , and J d is a d × d dimensional unit matrix (matrix only filled with ones).

Original: The parameter γ ∈ [0 , 1] can be regarded as a forgetting coefficient.
Revised: The parameter γ ∈ [0 , 1] can be regarded as a forgetting coefficient.

Original: Attention mechanism.
Revised: Attention mechanism.

Original: A mechanism of attention is trained to generate the next location of focus of attention, based on the current internal representation, i.e.,
Revised: A mechanism of attention is trained to generate the next location of focus of attention, based on the current internal representation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Since we aim at developing a purely task-driven mechanism of attention, parameters of the attention model are optimized to minimize the loss function
Revised: Since we aim at developing a purely task-driven mechanism of attention, parameters of the attention model are optimized to minimize the loss function

Original: [Equation]
Revised: [Equation]

Original: The optimal attention mechanism would unblur the regions that lead to the best performance with respect to the underlying vision task.
Revised: The optimal attention mechanism would unblur the regions that lead to the best performance with respect to the underlying vision task.

Original: Inference with NeVA.
Revised: Inference with NeVA.

Original: The task model is only used to generate the loss during training, and can be discarded at inference time.
Revised: The task model is only used to generate the loss during training, and can be discarded at inference time.

Original: The attention mechanism can be used iteratively, to infer the next position of the focus of attention, based on the updated agent’s state.
Revised: The attention mechanism can be used iteratively, to infer the next position of the focus of attention, based on the updated agent’s state.

Original:  Experiments
Revised:  Experiments

Original: The NeVA algorithm is used to generate scanpath of visual attention.
Revised: The NeVA algorithm is used to generate scanpath of visual attention.

Original: The plausibility of these scanpaths is measured through similarity metrics with human eye-tracking data.
Revised: The plausibility of these scanpaths is measured through similarity metrics with human eye-tracking data.

Original: We stress out that eye-tracking data is not used during the training of NeVA, but only for evaluation.
Revised: We stress out that eye-tracking data is not used during the training of NeVA, but only for evaluation.

Original: The performance of NeVA is compared with that of three unsupervised human attention models and three baselines.
Revised: The performance of NeVA is compared with that of three unsupervised human attention models and three baselines.

Original: Datasets.
Revised: Datasets.

Original: A collection of three well-established eye-tracking dataset is used in our study:
Revised: A collection of three well-established eye-tracking dataset is used in our study:

Original: MIT1003 [12] (1003 images, 15 subjects, free viewing condition), TORONTO [6] (120 images, 20 subjects, free viewing condition), and KOOTSTRA [15] (99 images, 31 subjects, free viewing condition).
Revised: MIT1003 [12] (1003 images, 15 subjects, free viewing condition), TORONTO [6] (120 images, 20 subjects, free viewing condition), and KOOTSTRA [15] (99 images, 31 subjects, free viewing condition).

Original: Metrics.
Revised: Metrics.

Original: Similarity with respect of human eye-tracking data is measured using two metrics.
Revised: Similarity with respect of human eye-tracking data is measured using two metrics.

Original: The string-edit distance (SED) [13] has been adapted in the domain of visual attention analysis to compare visual scanpaths [5, 9], after being properly converted to strings.
Revised: The string-edit distance (SED) [13] has been adapted in the domain of visual attention analysis to compare visual scanpaths [5, 9], after being properly converted to strings.

Original: Additionally, we define string-based time-delay embeddings (SBTDE) that computes the time delay embedding [1] in the string domain.
Revised: Additionally, we define string-based time-delay embeddings (SBTDE) that computes the time delay embedding [1] in the string domain.

Original: This metric better take into account the stochastic component of the attention process [17], and making the results more robust with respect to changes in both stimulus resolution and scanpath length.
Revised: This metric better take into account the stochastic component of the attention process [17], and making the results more robust with respect to changes in both stimulus resolution and scanpath length.

Original: Metrics are presented in two versions: Mean metrics are computed by simply averaging the scores with respect to all available subjects for a given image, while ScanPath Plausibility (SPP) metrics only considers the scanpath of minimal distance [8].
Revised: Metrics are presented in two versions: Mean metrics are computed by simply averaging the scores with respect to all available subjects for a given image, while ScanPath Plausibility (SPP) metrics only considers the scanpath of minimal distance [8].

Original: Competitors and baselines.
Revised: Competitors and baselines.

Original: For our experimental comparison, we restrict our comparison to unsupervised approaches, i.e., existing approaches that similarly to us do not use any eye-tracking data to train their models.
Revised: For our experimental comparison, we restrict our comparison to unsupervised approaches, i.e., existing approaches that similarly to us do not use any eye-tracking data to train their models.

Original: In particular, we include Constrained Levy Exploration (CLE) [3],
Revised: In particular, we include Constrained Levy Exploration (CLE) [3],

Original: Gravitational Eye Movements Laws (G-EYMOL)
Revised: Gravitational Eye Movements Laws (G-EYMOL)

Original: [22], Winner-take-all (WTA) [14].
Revised: [22], Winner-take-all (WTA) [14].

Original: CLE and WTA are based on saliency maps by [11].
Revised: CLE and WTA are based on saliency maps by [11].

Original: For all competitors, we use the python implementation provided by the authors in the original paper.
Revised: For all competitors, we use the python implementation provided by the authors in the original paper.

Original: Additionally, we define three baselines which help in better positioning the results.
Revised: Additionally, we define three baselines which help in better positioning the results.

Original: For the Random baseline, scanpaths are generated as sequences of random fixation points.
Revised: For the Random baseline, scanpaths are generated as sequences of random fixation points.

Original: For the Center baseline, subsequent fixations are sampled according to a center blob, as described in [12].
Revised: For the Center baseline, subsequent fixations are sampled according to a center blob, as described in [12].

Original: We finally regard the Human baseline as the gold standard, where we measure as each human is a good predictor for the remaining population on certain image.
Revised: We finally regard the Human baseline as the gold standard, where we measure as each human is a good predictor for the remaining population on certain image.

Original: NeVA versions.
Revised: NeVA versions.

Original: We test two different NeVA configurations.
Revised: We test two different NeVA configurations.

Original: NeVA C is based on a classification task.
Revised: NeVA C is based on a classification task.

Original: A ResNet that was trained on the CIFAR10 dataset [16] from the RobustBench library.
Revised: A ResNet that was trained on the CIFAR10 dataset [16] from the RobustBench library.

Original: NeVA R is based on a reconstruction-task model.
Revised: NeVA R is based on a reconstruction-task model.

Original: In this case, we train a denoising autoencoder on the CIFAR dataset using the implementation proposed in [24].
Revised: In this case, we train a denoising autoencoder on the CIFAR dataset using the implementation proposed in [24].

Original: For both cases, attention models is based on wide
Revised: For both cases, attention models is based on wide

Original: ResNets [20].
Revised: ResNets [20].

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: For each model and baseline, we generated scanpaths of length 10 , i.e., a sequence offixations.
Revised: For each model and baseline, we generated scanpaths of length 10 , i.e., a sequence offixations.

Original: Table 1 summarizes the results for all metrics and datasets.
Revised: Table 1 summarizes the results for all metrics and datasets.

Original: NeVA C is the best performing method in 11 / 12 metrics and the second-best in the remaining 1 .
Revised: NeVA C is the best performing method in 11 / 12 metrics and the second-best in the remaining 1 .

Original: NeVA R is the second best method in 7 / 12 metrics.
Revised: NeVA R is the second best method in 7 / 12 metrics.

Original: G-Eymol is the second-best method in 4 metrics, while CLE is the best method in 1 metric and the second-best method in 1 metric (same score as G-Eymol).
Revised: G-Eymol is the second-best method in 4 metrics, while CLE is the best method in 1 metric and the second-best method in 1 metric (same score as G-Eymol).

Original: Unlike competitors,
Revised: Unlike competitors,

Original: NeVA-based approaches are merely driven by their top-down signal (purely task-driven), and this allows us to directly compare the tasks involved.
Revised: NeVA-based approaches are merely driven by their top-down signal (purely task-driven), and this allows us to directly compare the tasks involved.

Original: In fact, we notice that classification tasks better explain human behavior over all three datasets.
Revised: In fact, we notice that classification tasks better explain human behavior over all three datasets.

Original: Moreover, for scanpath of length 10, the NeVA top-down approach produces better results than bottom-up counterparts, supporting the hypothesis that task guidance already emerges in the first phases (within 3 seconds) of visual explorations.
Revised: Moreover, for scanpath of length 10, the NeVA top-down approach produces better results than bottom-up counterparts, supporting the hypothesis that task guidance already emerges in the first phases (within 3 seconds) of visual explorations.

Original: The
Revised: The

Original: Center baseline did not perform much better than Random.
Revised: Center baseline did not perform much better than Random.

Original: This is a surprising result if we consider that a center blob can predict saliency very well [12, 4].
Revised: This is a surprising result if we consider that a center blob can predict saliency very well [12, 4].

Original: Human baseline, instead, outperforms all approaches by a large margin.
Revised: Human baseline, instead, outperforms all approaches by a large margin.

Original: This is particularly true in the case of the SPP versions of the metrics, where only the closest human scanpath is considered for the metric calculation.
Revised: This is particularly true in the case of the SPP versions of the metrics, where only the closest human scanpath is considered for the metric calculation.

Original: This result suggests that there is still large margin of improvement, especially in the development of personalized models of attention, which can take into account the large intra-population differences.
Revised: This result suggests that there is still large margin of improvement, especially in the development of personalized models of attention, which can take into account the large intra-population differences.

Original:  Discussion
Revised:  Discussion

Original: In an empirical study, we demonstrate that neural network models pre-trained for image classification or reconstruction can provide effective guidance for generating biologically-plausible visual scanpaths.
Revised: In an empirical study, we demonstrate that neural network models pre-trained for image classification or reconstruction can provide effective guidance for generating biologically-plausible visual scanpaths.

Original: Such scanpaths resemble human scanpaths, although they were neither explicitly trained for this purpose nor did we use eye movement data during training.
Revised: Such scanpaths resemble human scanpaths, although they were neither explicitly trained for this purpose nor did we use eye movement data during training.

Original: A future study should examine whether the similarity between human and artificial scanpaths can be improved by adding further biological constraints to neural networks.
Revised: A future study should examine whether the similarity between human and artificial scanpaths can be improved by adding further biological constraints to neural networks.

Original: Furthermore, the scanpaths generated by more complex task models need to be analyzed.
Revised: Furthermore, the scanpaths generated by more complex task models need to be analyzed.

Original: More complex task models could include object detectors, segmentation models, and visual transformers.
Revised: More complex task models could include object detectors, segmentation models, and visual transformers.

Original: Lastly, further work should investigate if NeVA can create highly task-specific scanpaths that resemble those of a human experts (i.e., by using a network trained to detect tumors to generate scanpaths for a medical application).
Revised: Lastly, further work should investigate if NeVA can create highly task-specific scanpaths that resemble those of a human experts (i.e., by using a network trained to detect tumors to generate scanpaths for a medical application).

Original: These scanpaths could be used to constrain the attention of neural networks to relevant image regions and thus improve the computational load.
Revised: These scanpaths could be used to constrain the attention of neural networks to relevant image regions and thus improve the computational load.

Original: A Example scanpaths
Revised: A Example scanpaths

Original: To provide a better intution about the outcome of the NeVA algorithm here we provide some illustrations.
Revised: To provide a better intution about the outcome of the NeVA algorithm here we provide some illustrations.

Original: Figure 2 shows example scanpaths generated by the NeVA algorithm.
Revised: Figure 2 shows example scanpaths generated by the NeVA algorithm.

Original: The first row shows the original stimulus, the second row shows the foveation heatmaps and fixations and the last row shows the internal representation after the last fixation
Revised: The first row shows the original stimulus, the second row shows the foveation heatmaps and fixations and the last row shows the internal representation after the last fixation