Comparing and Predicting Eye-tracking Data of M andarin and C antonese (2023)

Abstract: Eye-tracking data in Chinese languages present unique challenges due to the non-alphabetic and unspaced nature of the Chinese writing systems. This paper introduces the first deeply-annotated joint Mandarin-Cantonese eye-tracking dataset, from which we achieve a unified eye-tracking prediction system for both language varieties. In addition to the commonly studied first fixation duration and the total fixation duration, this dataset also includes the second fixation duration, expressing fixation patterns that are more relevant to higher-level, structural processing. A basic comparison of the features and measurements in our dataset revealed variation between Mandarin and Cantonese on fixation patterns related to word class and word position. The test of feature usefulness suggested that traditional features are less powerful in predicting the second-pass fixation, to which the linear distance to root makes a leading contribution in Mandarin. In contrast, Cantonese eye-movement behavior relies more on word position and part of speech.

## introduction eye-tracking has quickly become one of the most popular methodologies in psycholinguistic studies, as it allows researchers to measure people's real-time processing efforts during a reading task @xcite . consequently, more computational models have been proposed to predict eye-fixation patterns in english and many other languages @xcite @xcite . chinese languages, being non-alphabetic, are considered unique in eye-tracking research, mainly due to the unspaced nature of the writing conventions, the visual complexity of the characters, and the abundance of homophonic and homographic characters @xcite . the computational modeling of chinese eye-movement patterns is still relatively limited, although several traditional psycholinguistic models have been proposed to measure chinese reading times @xcite @xcite . such models have focused on factors such as word frequency, word length, and word predictability but have not considered syntactic and semantic processes that may have an equally decisive influence on eye-movement behaviors. to fill such research gaps, this paper first introduces a deeply-annotated eye-tracking dataset that covers mandarin texts in simplified characters and cantonese texts in traditional characters, thus representing two demographically important language varieties. based on this joint dataset, we implemented a series of statistical tests to investigate the inter-linguistic variance from the perspective of fixation durations. furthermore, we propose a feature-rich prediction model of basic eye-tracking measurements in chinese, in addition to an ablation study of the usefulness of features. our predictors include both traditional and new features, such as syntactic features, local lexical semantic features, and contextual semantic representations. we believe that comparing these features will further broaden our understanding of the differences between mandarin and cantonese. the contributions of the present study are as follows: ## related work as eye-tracking data are closely linked to realtime cognitive processes, they can reveal the automatic operations in our brains that are related to different linguistic modules, such as lexical access @xcite , syntactic processing @xcite , semantic processing @xcite , and pragmatic competence @xcite . regarding the modeling of fixation patterns, previous research has highlighted the close relationship between eye-tracking measurements and certain word properties, including word position (just and carpenter, 1980), word frequency @xcite , word predictability @xcite , and word length @xcite . fixation on a particular word is also sensitive to the cognitive load from the previous word (known as a spill-over effect) @xcite . 1 in addition to these traditional lexical features, morpho-syntactic features, such as part-of-speech categories (pos) and syntactic dependency, also impact fixation patterns. pos have been demonstrated to influence the number of fixations and the fixation duration @xcite . concerning syntactic dependency, previous studies indicated that cognitive loads from syntactic structure lead to increased refixation probability and duration @xcite , which is mainly related to the second fixation duration (sfd) in this paper and partially reflected on the total fixation duration (tfd). previous research also reported that the sensitivity of first-pass processing to the syntactic agreement increases the first fixation duration @xcite , although there 1 according to some studies, another factor affecting fixations is the semantic relatedness of a word with its context, which can be measured via distributional semantic models @xcite @xcite . however, the evidence for the role of semantic relatedness in predicting reading times and eye fixations is controversial @xcite . regarding the distinctiveness of the chinese writing system, most studies have supported the view that words and characters are equally salient units in the cognitive processing of texts written in chinese characters, as both word properties and character properties influence reading-time measurements and eye-movement behaviors @xcite . following this assertion, the word-level features widely applied in the readingtime modeling of alphabetic languages are equally applied in chinese-specific research. features related to higher-level processing, such as syntactic properties, are also considered in research on chinese language processing @xcite @xcite . however, previous studies using syntactic properties have mainly focused on syntactic complexity and the grammatical function of a word without linking the syntactic dependency of the entire sentence to eye-movement modeling. ## dataset this section introduces our eye-tracking dataset's construction procedures and annotation structure. 2 we then present the results of inter-variety comparisons regarding basic eye-tracking measurements in the next section. ## methodology this section introduces the features and the regressors used in the prediction of eye-tracking measurements, derived from the results of the cross-variety comparison drawn from both psycholinguistic and computational studies. ## conclusions in this paper, we introduced an extensively annotated dataset of mandarin and cantonese eyetracking data and shed light on their differences by features, such as word formation, word class, and word order. we also proposed a prediction system of fixation behaviors accompanied by new features from different modules, such as dependency features, the orthographic neighborhood, and gpt word embeddings, which were introduced with the goal of the computational prediction of chinese eye-tracking data. based on a comparison of the regressor performance under different feature compositions, we investigated the usefulness of gpt vectors and linguistic features in reducing prediction errors. the results highlighted the effectiveness of our newly introduced features in modeling fixation patterns in representative chinese language varieties and the importance of word order, part-of-speech, and syntax in addressing how mandarin and cantonese differ in language comprehension. the findings in our study identify a few possible topics for future studies on language processing and regional syntactic variation of chinese languages, such as how the syllabic structure, the visual complexity of different writing systems, pronominal resolution, syntactic relations, word order interact with gazing patterns and reading times of chinese language speakers, especially for native speakers of different varieties. in addition to the varieties we studied here, we also plan to enlarge the dataset by including mandarin processed through traditional chinese characters, which is the standard system used in taiwan. finally, for future psychological and computational modeling studies, possible refinements of the representations in our experiment could be features targeting orthographic complexity and lexical ambiguity.