S can DL : A Diffusion Model for Generating Synthetic Scanpaths on Texts (2023)

Abstract: Eye movements in reading play a crucial role in psycholinguistic research studying the cognitive mechanisms underlying human language processing. More recently, the tight coupling between eye movements and cognition has also been leveraged for language-related machine learning tasks such as the interpretability, enhancement, and pre-training of language models, as well as the inference of reader- and text-specific properties. However, scarcity of eye movement data and its unavailability at application time poses a major challenge for this line of research. Initially, this problem was tackled by resorting to cognitive models for synthesizing eye movement data. However, for the sole purpose of generating human-like scanpaths, purely data-driven machine-learning-based methods have proven to be more suitable. Following recent advances in adapting diffusion processes to discrete data, we propose ScanDL, a novel discrete sequence-to-sequence diffusion model that generates synthetic scanpaths on texts. By leveraging pre-trained word representations and jointly embedding both the stimulus text and the fixation sequence, our model captures multi-modal interactions between the two inputs. We evaluate ScanDL within- and across-dataset and demonstrate that it significantly outperforms state-of-the-art scanpath generation methods. Finally, we provide an extensive psycholinguistic analysis that underlines the model’s ability to exhibit human-like reading behavior. Our implementation is made available at https://github.com/DiLi-Lab/ScanDL.

## introduction as human eye movements during reading provide both insight into the cognitive mechanisms involved in language processing @xcite and information about the key properties of the text @xcite , they have been attracting increasing attention from across fields, including cognitive psychology, experimental and computational psycholinguistics, and computer science. as a result, computational models of eye movements in reading experienced an upsurge over the past two decades. the earlier models are explicit computational cognitive models designed for fundamental research with the aim of shedding light on i) the mechanisms underlying human language comprehension at different linguistic levels and ii) the broader question of how the human language processing system interacts with domain-general cognitive mechanisms and capacities, such as working memory or visual attention @xcite @xcite . more recently, traditional and neural machine learning (ml) approaches have been adopted for the prediction of human eye movements @xcite @xcite @xcite which, in contrast to cognitive models, do not implement any psychological or linguistic theory of eye movement control in reading. ml models, in turn, exhibit the flexibility to learn from and adapt to any kind of reading pattern on any kind of text. within the field of ml, researchers have not only begun to create synthetic scanpaths but also, especially within nlp, to leverage them for different use cases: the interpretability of language models (lms) @xcite @xcite , enhancing the performance of lms on downstream tasks @xcite @xcite , and pre-training models for all kinds of inference tasks concerning reader-and text-specific proper-ties (e.g., assessing reading comprehension skills or l2 proficiency, detecting dyslexia, or judging text readability) @xcite @xcite @xcite . for these nlp use cases, the opportunity to generate large amounts of synthetic scanpaths is crucial for two reasons: first, real human eye movement data is scarce, and its collection is resource intensive. second, relying on real human scanpaths entails the problem of not being able to generalize beyond the respective dataset, as gaze recordings are typically not available at inference time. synthetic scanpaths resolve both issues. however, generating synthetic scanpaths is not a trivial task, as it is a sequence-to-sequence problem that requires the alignment of two different input sequences: the word sequence (order of words in the sentence), and the scanpath (chronological order of fixations on the sentence). in this paper, we present a novel discrete sequence-to-sequence diffusion model for the generation of synthetic human scanpaths on a given stimulus text: scandl, scanpath diffusion conditioned on language input (see figure 1 ). scandl leverages pre-trained word representations for the text to guide the model's predictions of the location and the order of the fixations. moreover, it aligns the different sequences and modalities (text and eye gaze) by jointly embedding them in the same continuous space, thereby capturing dependencies and interactions between the two input sequences. the contributions of this work are manifold: we (i) develop scandl, the first diffusion model for simulating scanpaths in reading, which outperforms all previous state-of-the-art methods; we (ii) demonstrate scandl's ability to exhibit humanlike reading behavior, by means of a bayesian psycholinguistic analysis; and we (iii) conduct an extensive ablation study to investigate the model's different components, evaluate its predictive capabilities with respect to scanpath characteristics, and provide a qualitative analysis of the model's decoding process. ## problem setting consider a scanpath f r @xmath0, . . . , f n ⟩, which represents a sequence of n fixations generated by reader r while reading sentence w. here, f j denotes the location of the j th fixation represented by the linear position of the fixated word within the sentence w (word index). the goal is to find a model that predicts a scanpath f given sentence w. we evaluate the model by computing the mean normalized levenshtein distance (nld) @xcite between the predicted and the ground truth human scanpaths. note that several readers r can read the same sentence w. in the following, we will denote the scanpath by f instead of f r w if the reader or sentence is unambiguous or the (predicted) scanpath is not dependent on the reader. ## scandl inspired by continuous diffusion models for text generation @xcite , we propose scandl, a diffusion model that synthesizes scanpaths conditioned on a stimulus sentence. ## blocks, with 8 attention heads and hidden dimension @xmath1. an extra linear layer projects the pre-trained bert embedding emb bert to the hidden dimension d. the maximum sequence length is 128, and the number of diffusion steps is @xmath2. we use a sqrt noise schedule @xcite to sample β @xmath3, where @xmath4.0001 is a small constant corresponding to the initial noise level. to sample the noising step @xmath5, 1, . . . , t , we employ importance sampling as defined by nichol and dhariwal (2021). ## experiments to evaluate the performance of scandl against both cognitive and neural scanpath generation methods, we perform a within-and an acrossdataset evaluation. all models were implemented in pytorch @xcite , and trained for 80,000 steps on four nvidia geforce rtx 3090 gpus. for more details on the training, see appendix a. our code is publicly available at https: //github.com/dili-lab/scandl . ## discussion the experimental results show that our model establishes a new state-of-the-art in generating humanlike synthetic eye movement patterns across all investigated evaluation scenarios, including withinand across-dataset settings. indeed, the similarity of the model's synthetic scanpaths to a human scanpath even exceeds the similarity between two scanpaths generated by two different human readers on the same stimulus. this indicates that the model learns to mimic an average human reader, abstracting away from reader-specific idiosyncrasies. 6 interestingly, the model's performance is better in the new reader than in both the new sentence and new reader/new sentence setting, which stands in stark contrast to previous research which identified the generalization to novel readers as the 6 this is further supported by the nld not changing when computed as average nld between a scandl scanpath and all human scanpaths on the same sentence in the test set. main challenge @xcite . furthermore, scandl's sota performance in the across-dataset setting, attaining parity with the human baseline, corroborates the model's generalizability. this generalizability is further underlined by the model emulating the variability in reading patterns observed in the human data, even when being evaluated across-dataset. the omission of the positional embedding and the pre-trained bert embedding in the ablation study highlights their importance -the fact that discarding them yields a worse performance than omitting the sentence condition, in which case the model still receives the positional embedding, stresses the importance of sequential information, which is lost to a transformer model if not explicitly provided. moreover, removing the sentencecondition also emphasizes the importance of the linguistic information contained in the sentence. overall, the ablation study emphasizes the importance of modeling scanpath prediction as a dualnature problem. in contrast to cognitive models, scandl was not designed to exhibit the same psycholinguistic phenomena as human readers. however, our psycholinguistic analysis demonstrates that scandl nevertheless captures the key phenomena observed in human readers which psycholinguistic theories build on. even more, in contrast to the cognitive models, it appears to overfit less to certain reading patterns. these findings not only emphasize the high quality of the generated data, but also open the possibility to use scandl when designing psycholinguistic experiments: the experimental stimuli can be piloted by means of simulations with scandl to potentially detect unexpected patterns or confounds and address them before conducting the actual experiment with human readers. further, we can use scandl for human-centric nlg evaluations using synthetic scanpaths. ## conclusion we have introduced a new state-of-the-art model for scanpath generation called scandl. it not only resolves the two major bottlenecks for cognitively enhanced and interpretable nlp, data scarcity and unavailability at inference time, but also promises to be valuable for psycholinguistics by producing high quality human-like scanpaths. further, we have extended the application of diffusion models to discrete sequence-to-sequence problems.