A Unified Framework for Synaesthesia Analysis (2023)

Abstract: Synaesthesia refers to the description of perceptions in one sensory modality through concepts from other modalities. It involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought and action, which makes understanding it challenging. As a means of cognition, synaesthesia is rendered by more than sensory modalities, cue and stimulus can also play an important role in expressing and understanding it. In addition, understanding synaesthesia involves many cognitive efforts, such as identifying the semantic relationship between sensory words and modalities. Therefore, we propose a unified framework focusing on annotating all kinds of synaesthetic elements and fully exploring the relationship among them. In particular, we introduce a new annotation scheme, including sensory modalities as well as their cues and stimuli, which facilitate understanding synaesthetic information collectively. We further design a structure generation model to capture the relations among synaesthetic elements and generate them jointly. Through extensive experiments, the importance of proposed dataset can be verified by the statistics and progressive performances. In addition, our proposed model yields state-of-the-art results, demonstrating its effectiveness.

## introduction synaesthesia, based on the greek roots 'syn' (together) and 'aisthesia' (perception), describes a situation in which perceptions in different sensory modalities are associated in both perceptual experiences and verbal expressions @xcite @xcite . synaesthesia in verbal expressions is a language usage whereby lexical items in one sensory modality are employed to describe perceptions in another sensory modality @xcite 2020). for instance, as shown in figure 1 , the gustatory adjective "sweet" can be used to describe an auditory perception, as in the phrase "said in a sweet tone". there are extensive studies on linguistic synaesthesia from various perspectives @xcite @xcite . nevertheless, synaesthesia has received little attention in natural language processing. as one of the initial study, @xcite constructed a humanannotated chinese synaesthesia dataset, where sensory words, original and synaesthetic sensory modalities are annotated. they further designed a pipeline system to identify the sensory word, and to detect the original and synaesthetic sensory modalities for the sensory word. as a means of cognition and communication, synaesthesia is rendered by more than sensory modalities, cue and stimulus can also contribute to synaesthetic conceptualization. cue and stimulus are the trigger words or expressions that cause the synaesthetic usages. as shown in figure 1 , since the cue "tone" and the stimulus "said" are both auditory words, it is easy to find that "sweet" is transferred from taste to hearing. in addition, understanding synaesthesia involves many cognitive efforts, such as identifying the semantic re-lationship between sensory words and modalities, as well as interpreting sensory modalities with the cue and the stimulus, which might be difficult for computers to deal with. in this study, we propose a unified framework to address the above challenges by annotating all kinds of synaesthetic elements and fully exploring the relationship among them. in particular, we firstly extend the previous dataset by collecting more samples from social media. we then propose a new annotation framework, including sensory modalities as well as their cues and stimuli, which facilitate understanding synaesthesia information and help to improve the performance of automatic synaesthesia comprehension systems. furthermore, we propose a structure generation model to capture the relations among synaesthetic elements, and generate all elements jointly. afterwards, we employ a structure acquisition and composition based framework to improve the generation model by capturing the interdependency among synaesthetic elements. the statistics show the potential and usefulness of the new annotation scheme. in addition, the experimental results demonstrate the effectiveness of the proposed model.we release our dataset named csyna at https://github.com/dinoksaur/csyna . ## related works studies on linguistic synaesthesia from a linguistic perspective focus on the directionality patterns and underlying mechanisms for synaesthetic transfers between different modalities. note that "synaesthesia" and "synesthesia" are used interchangeably in the literature. for consistency, we use "synaesthesia" in this paper. for instance, previous studies @xcite @xcite found that the transfers of linguistic synaesthesia conformed to certain patterns, rather than mapping randomly. in terms of the mechanisms underlying synaesthetic transfers, @xcite and @xcite have suggested that linguistic synaesthesia is grounded in multiple mechanisms. in addition, @xcite @xcite focus on identifying linguistic synaesthetic expressions in natural languages. however, their studies are conducted by semiautomatic methods with lots of manual strategies. there are no comprehensive computational models with automatic synaesthesia detection em-ployed in previous methods. recently @xcite constructed a human-annotated chinese synaesthesia dataset. they further proposed a pipeline system to identify sensory words, and to detect the original and synaesthetic sensory modalities. however, the annotated samples in their dataset is limited, and cue and stimulus are missing in their annotation scheme. therefore, we propose a unified framework focusing on annotating all kinds of synaesthetic elements and fully exploring the relationship among them. in particular, we propose a new annotation scheme, including sensory modalities as well as their cues and stimuli, which are helpful for understanding synaesthesia information. we further employ a structure generation model to fully explore the relations between synaesthetic elements and to generate them jointly. ## data annotation and analysis in this section, we first give a new scheme of synaesthesia. we then show the collection and annotation process of the dataset. after that, we give some fundamental statistics and analyses. ## synaesthetic structure generation given a sentence, we employ a structure generation model to generate all the synaesthetic ele-ments from a sentence. as shown in figure 3 , we firstly convert all the synaesthetic elements into a tree structure. we then employ a tree generation model to explore the relations among synaesthetic elements and generate them jointly. afterwards, we utilize a structure acquisition and composition based framework to improve the structure generation model by capturing the interdependency among synaesthetic elements. in this section, we give the description of synaesthetic tree construction and the structure tree generation model, and then discuss the structure acquisition and composition framework in the next section. ## structure acquisition and composition since the alignment between sentences and synaesthetic elements, and the interdependency among synaesthetic elements are very important for synaesthesia analysis, we then propose a novel strategy to improve the structure generation model by addressing the above issues. as shown in figure 4 , we firstly decompose the original task into several subtasks. each subtask corresponds to mapping the natural language sentence to one or more synaesthetic elements. afterwards, we feature a prompt-based learning strategy to separately acquire the structural knowledge of subtasks and employ the learned knowledge to tackle the main task, i.e., generating all the synaesthetic elements. in the structure acquisition stage, we train the model with all the subtasks in a multitask learning manner; in the structure composition stage, we fine-tune the model with the main task to combine the acquired structural knowledge of subtasks and learn the dependency between them. ## experiments in this section, we conduct various experiments to investigate the effectiveness of the proposed method on the synaesthetic elements generation task. in addition, we give several analyses and discussions to show the importance of the proposed new dataset. ## conclusion synaesthesia refers to the description of perceptions in one sensory modality through concepts from other modalities. it involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought and action, which makes understanding it challenging and re-warding. in this study, we give a comprehensive study on the synaesthesia analysis from a computational perspective. in particular, we firstly introduce a new annotation framework, including annotating the sensory modality as well as their cues and stimuli, which facilitates understanding the synaesthesia information. we further design a structure generation model to fully explore the relations among synaesthetic elements and to generate them jointly. the statistics show the potential and usefulness of the proposed new dataset. in addition, the experimental results demonstrate the effectiveness of the proposed model.