G enerating I mage D escriptions via S equential C ross- M odal A lignment G uided by H uman G aze (2020)

Abstract: When speakers describe an image, they tend to look at objects before mentioning them. In this paper, we investigate such sequential cross-modal alignment by modelling the image description generation process computationally. We take as our starting point a state-of-the-art image captioning system and develop several model variants that exploit information from human gaze patterns recorded during language production. In particular, we propose the first approach to image description generation where visual processing is modelled sequentially. Our experiments and analyses confirm that better descriptions can be obtained by exploiting gaze-driven attention and shed light on human cognitive processes by comparing different ways of aligning the gaze modality with language production. We find that processing gaze data sequentially leads to descriptions that are better aligned to those produced by speakers, more diverse, and more natural—particularly when gaze is encoded with a dedicated recurrent component.

## introduction describing an image requires the coordination of different modalities. there is a long tradition of cognitive studies showing that the interplay between language and vision is complex. on the one hand, eye movements are influenced by the task at hand, such as locating objects or verbally describing an image @xcite . on the other hand, visual information processing plays a role in guiding linguistic production (e.g., @xcite . such cross-modal coordination unfolds sequentially in the specific task of image description (coco and keller, 2012)-i.e., objects tend to be looked at before being mentioned. yet, the temporal alignment between the two modalities is not straightforward @xcite figure 1 : in our approach, an image captioning model is fed with a sequence of masked images encoding the gaze fixations of a single human speaker during language production. this diagram is a toy illustration. in this paper, we follow up on these findings and investigate cross-modal alignment in image description by modelling the description generation process computationally. we take a state-of-the-art system for automatic image captioning @xcite and develop several model variants that exploit information derived from eye-tracking data. to train these models, we use a relatively small dataset of image descriptions in dutch (didec; @xcite that includes information on gaze patterns collected during language production. we hypothesise that a system that encodes gaze data as a proxy for human visual attention will lead to better, more human-like descriptions. in particular, we propose that training such a system with eye-movements sequentially aligned with utterances (see figure 1 ) will produce descriptions that reflect the complex coordination across modalities observed in cognitive studies. we develop a novel metric that measures the level of semantic and sequential alignment between descriptions and use it in two ways. first, we analyse cross-modal coordination in the didec data, finding that the product of content and sequentiality better captures cross-modal correlations than content alone. second, we test whether our models generate captions that capture sequential alignment. our experiments show that exploiting gaze-driven attention helps enhance image caption generation, and that processing gaze patterns sequentially results in descriptions that are better aligned with those produced by speakers, as well as being more diverse-both in terms of variability per image and overall vocabulary-particularly when gaze is encoded with a dedicated recurrent component that can better capture the complexity of the temporal alignment across modalities. our data and code are publicly available at https: //github.com/dmg-illc/didec-seq-gen. overall, this work presents the first computational model of image description generation where both visual and linguistic processing are modelled sequentially, and lends further support to cognitive theories of sequential cross-modal coordination. we take as our starting point the model by @xcite for two main reasons: first, it is among the best-performing architectures on standard image captioning benchmarks; second, its underlying idea (i.e., bottom-up and top-down attention) is explicitly inspired by human visual attention mechanisms @xcite , which makes it suitable for investigating the impact of adding human gaze information. eye tracking in computer vision, human eye movements collected with eye-tracking methods have been exploited to model what is salient in an image or video for object detection @xcite , image classification @xcite , image segmentation @xcite , region labelling @xcite , and action detection @xcite . more relevant for the present study, gaze has also been used in automatic description generation tasks, such as video frame captioning @xcite and image captioning @xcite @xcite . in all these approaches, gaze data from different participants is aggregated into a static saliency map to represent an abstract notion of saliency. this aggregated gaze data is used as supervision to train models that predict generic visual saliency. in contrast, in our approach, we model the production process of a single speaker by directly inputting information about where that speaker looks at during description production, and compare this to the aggregation approach. in addition, we exploit the sequential nature of gaze patterns, i.e., the so-called scanpath, and contrast this with the use of static saliency maps. gaze scanpaths have been used in nlp for diverse purposes: for example, to aid part-of-speech tagging @xcite and chunking @xcite ; to act as a regulariser in sequence classification tasks @xcite ; as well as for automatic word acquisition @xcite and reference resolution @xcite . to our knowledge, the present study is the first attempt to investigate sequential gaze information for the specific task of image description generation. ## data we utilise the dutch image description and eye-tracking corpus (didec; @xcite . in particular, we use the data collected as part of the description-view task in didec, where participants utter a spoken description in dutch for each image they look at. the gaze of the participants is recorded with an smi red 250 eyetracking device while they describe an image. over-all, didec consists of 4604 descriptions in dutch (15 descriptions per image on average) for 307 ms coco images @xcite . for each description, the audio, textual transcription, and the corresponding eye-tracking data are provided. ## evaluation measures we propose a novel metric to quantify the degree of both semantic and sequential alignment between two sentences. in our study, this metric will be leveraged in two ways: (1) to analyse cross-modal coordination in the didec data (section 5) and (2) to evaluate our generation models (section 7). for context, we first briefly review several existing metrics for automatic image captioning. image captioning metrics image caption generation is evaluated by assessing some kind of similarity between the generated caption and one or more reference captions (i.e., those written by human annotators). one of the most commonly used metrics is cider @xcite , which (a) computes the overlapping n-grams between the generated caption and the entire set of reference sentences for a given image, and (b) downweighs n-grams that are frequent in the entire corpus via tf-idf scores. thus-regarding semantics and sequentiality-cider scores can be affected by word order permutations, but not by the relative position of words in the entire caption nor by the presence of different but semantically similar words. other metrics such as bleu (which looks at n-gram precision; @xcite and rouge-l (which considers n-gram recall; @xcite suffer from comparable limitations. meteor @xcite and spice @xcite also make use of n-grams (or tuples in a scene's graph, in the case of spice) and take into account semantic similarity by matching synonyms using wordnet @xcite . this allows for some flexibility, but can be too restrictive to grasp overall semantic similarity. to address this, @xcite proposed using wmd, which builds on word2vec embeddings @xcite ; more recently, several metrics capitalising on contextual embeddings @xcite were proposed, such as bertscore @xcite and mover-score @xcite . however, these metrics neglect the sequential alignment of sentences. 3 ssd we propose semantic and sequential distance (ssd), a metric which takes into account both semantic similarity and the overall relative order of words. regarding the latter, ssd is related to ordering-based sequence similarity (oss; @xcite , a measure used by @xcite to compare sequences of categories representing gaze patterns. 4 given two sequences of words, i.e., a generated sentence g and a reference sentence r, ssd provides a single positive value representing the overall dissimilarity between g and r: the closer the value to 0, the higher the similarity between the two sentences (note that the value is unbounded). this single value is the average of two terms, gr and rg, which quantify the overall distance between g and r-the sum of their cosine (cos) and positional (pos) distance-from g to r and from r to g, respectively. the equation for gr is given below: ## cross-modal coordination analysis to empirically motivate our generation models, as a preliminary experiment we investigate the level of coordination between visual attention and linguistic production in the didec dataset. in particular, we test whether scanpath similarity and sentence similarity are correlated and whether taking into account the sequential nature of the two modalities results in higher cross-modal alignment. we transform gaze data into time-ordered sequences of object labels, i.e., scanpaths, (e.g., @xmath0, 'person', 'cat', 'table') using the annotations of object bounding boxes in the ms coco image dataset. on average, scanpaths have a length of 23.4 object labels. as for captions, we simply take the full sentences and treat them as sequences of words (e.g., @xmath1. descriptions contain an average of 12.8 tokens. order-sensitive analysis (sequential) for each image, we take the set of produced descriptions and compute all pairwise similarities by using ssd (see section 4). similarly, we take the corresponding scanpaths and compute all pairwise similarities by using oss @xcite . we then calculate spearman's rank correlation (onetailed) between the two similarity lists. this way, we obtain a correlation coefficient and p-value for each of the 307 images in the dataset. ## models the starting point for our models is the one by @xcite . 5 the main aspect that distinguishes this model from other image captioning systems is the use of faster r-cnn @xcite as image encoder, which identifies regions of the image that correspond to objects and are therefore more salient-the authors refer to this type of saliency detection as "bottom-up attention". each object region i is transformed into an image feature vector v i . the set of region vectors {v 1 , . . . , v k } is utilised in two ways by two lstm modules: the first lstm takes as input the mean-pooled image feature v (i.e., the mean of all salient regions) at each time step, concatenated with the two standard elements of a language model, i.e., the previous hidden state and an embedding of the latest generated word. the hidden state of this first lstm is then used by an attention mechanism to weight the vectors in {v 1 , . . . , v k }-the authors refer to this kind of attention as "top-down". finally, the resulting weighted average feature vector vt is given as input to the second lstm module, which generates the caption one word at a time. note that the set of region vectors {v 1 , . . . , v k } and the meanpooled vector v are constant over the generation of a caption, while the weights over {v 1 , . . . , v k } and hence the weighted average feature vector vt do change dynamically at each time step since they are influenced by the words generated so far. we take the original model as our baseline and modify it to integrate visual attention defined by gaze behaviour. in particular, we replace the meanpooled vector v by a gaze vector g computed from masked images representing fixation patterns as explained in section 3. we do not directly modify the set of object regions {v 1 , . . . , v k } present in the original model (i.e., bottom-up attention is still present in our proposed models). however, the top-down attention weights learned by the models are influenced by the gaze patterns given as input. concretely, we test the following model conditions: for the three gaze models, we also considered a version where v is concatenated with g or g t as appropriate, rather than being replaced by the gaze vectors. since they did not bring in better results, we do not discuss them further in the paper. ## experiments we experiment with the proposed models using the didec dataset and report results per model type. ## analysis this section presents an analysis of the descriptions generated by the models on the test set (446 descriptions). we focus on one single run per model. ## conclusions we tackled the problem of automatically generating an image description from a novel perspective, by modelling the sequential visual processing of a speaker concurrently with language production. our study shows that better descriptions-i.e., more aligned with speakers' productions in terms of content and order of words-can be obtained by equipping models with human gaze data. moreover, this trend is more pronounced when gaze data is fed sequentially, in line with cognitive theories of sequential cross-modal alignment (e.g., @xcite . our study was conducted using the dutch language dataset didec @xcite , which posed the additional challenges of dealing with a small amount of data and a low resource language. we believe, however, that there is value in conducting research with languages other than english. in the future, our approach and new evaluation measure could be applied to larger eyetracking datasets, such as the english dataset by @xcite . since different eye-tracking datasets tend to make use of different gaze encodings and formats, the amount of pre-processing and analysis steps required to apply our method to other resources was beyond the scope of this paper. we leave testing whether the reported pattern of results holds across different languages to future work. despite the challenges mentioned above, our experiments show that a state-of-art image captioning model can be effectively extended to encode cognitive information present in human gaze behaviour. comparing different ways of aligning the gaze modality with language production, as we have done in the present work, can shed light on how these processes unfold in human cognition. this type of computational modelling could help, for example, study the interaction between gaze and the production of filler words and repetitions, which we have not investigated in detail. taken together, our results open the door to further work in this direction and support the case for computational approaches leveraging cognitive data. alfred l yarbus. 1967. eye movements during perception of complex objects. in eye movements and vision, pages 171-211. springer. youngjae yu, jongwook choi, yeonhwa kim, kyung yoo, sang-hun lee, and gunhee kim. 2017. supervising neural attention models for video captioning by human gaze data. conference on computer vision and pattern recognition (cvpr), pages 6119-6127. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in international conference on learning representations. wei zhao, maxime peyrard, fei liu, yang gao, christian m meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 563-578.