C hinese Synesthesia Detection: New Dataset and Models (2022)

Abstract: In this paper, we introduce a new task called synesthesia detection, which aims to extract the sensory word of a sentence, and to predict the original and synesthetic sensory modalities of the corresponding sensory word. Synesthesia refers to the description of perceptions in one sensory modality through concepts from other modalities. It involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought and action, which makes it become a bridge between figurative linguistic phenomenon and abstract cognition, and thus be helpful to understand the deep semantics. To address this, we construct a large-scale human-annotated Chinese synesthesia dataset, which contains 7,217 annotated sentences accompanied by 187 sensory words. Based on this dataset, we propose a family of strong and representative baseline models. Upon these baselines, we further propose a radical-based neural network model to identify the boundary of the sensory word, and to jointly detect the original and synesthetic sensory modalities for the word. Through extensive experiments, we observe that the importance of the proposed task and dataset can be verified by the statistics and progressive performances. In addition, our proposed model achieves state-of-the-art results on the synesthesia dataset.

## introduction synesthesia refers to the association of perceptions in both perceptual experiences and language usages @xcite . synesthesia in perceptions, namely neurological synesthesia, describes a special perceptual condition for specific people who can perceive colors from black-printed letters, touch sounds, taste shapes, and so forth @xcite . synesthesia in language usages, named linguistic synesthesia alternatively, involves lexical items in one sensory modality to describe perceptions in another sensory modality @xcite . for instance, as shown in figure 1 , the gustatory adjective "甜蜜蜜" (sweet) can be used to describe an auditory perception, as in the phrase "甜蜜蜜的 语调" (a sweet tone). different from extensive studies on synesthesia in neurological and linguistic areas, synesthesia has received little attention in natural language processing (nlp). one of the related topics in nlp is metaphor detection, which aims at identifying metaphorical expressions using computational models @xcite @xcite . that is, synesthesia involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought. naturally, synesthesia can bridge the gap between figurative linguistic phenomenon and abstract cognition on deep semantics. thus, it may help us understand figurative methods, the cause of commonsense, and the latent logic of natural language generation in a more cognitive way. however, synesthesia involves both the source and target domains in sensory modalities, while metaphor usually involves only the source domain in sensory modality @xcite . therefore, detecting synesthesia has its unique significance which is different from identifying metaphors. in this study, we introduce a new task called synesthesia detection for deep analysis of synesthesia using computational models. as shown in figure 1 , synesthesia detection aims to extract the sensory word, and to predict the original and synesthetic sensory modalities of the corresponding sensory word. there are five sensory modali-ties @xcite @xcite including touch, taste, smell, vision, and hearing in this study, and sensory word is an adjective word which expresses sensory perceptions in a sentence. "甜蜜蜜" (sweet) is the sensory word in the example. in addition, synesthesia is the mapping of sensory modalities for sensory words from their original domains to their synesthetic domains. the original sensory modality in the above example is taste, and the synesthetic sensory modality is hearing. meanwhile, we focus on chinese synesthesia detection in this study. different from english, chinese is an ideographic language featured by no word delimiter between words in written. furthermore, not only words and characters can express specific meanings in chinese, but also radicals are important carriers of semantics @xcite . as shown in figure 1 , a radical is often related to a specific concept and a specific sensory modality, i.e., the tongue ('舌' of '甜') for taste, and spoken language ('讠' of '语' and'调') for hearing. in this study, we make the following efforts to advance chinese synesthesia detection: first, we construct a mandarin chinese synesthesia dataset. specifically, we extract the sensory words from each sentence. we then annotate the original and synesthetic sensory modalities of the corresponding sensory word. there are 187 sensory words and 7,217 synesthetic sentences in the dataset, where visual adjectives, tactile adjectives, and gustatory adjectives are the top three lexical types. second, we establish a family of solid and representative baselines, including bilstm+crf, bert+crf, sr-bilstm, and pf-bert, to extract the sensory word and to automatically detect the original and synesthetic sensory modalities of the sensory word. upon these baselines, we further propose a radical-based neural network model to identify the sensory word's boundary and jointly classify the original and synesthetic sensory modalities. the experimental results demonstrate the effectiveness of the proposed model as the state-ofthe-art model for our constructed dataset. in summary, the contributions of this paper include: ## data annotation and analysis in this section, we first give some preliminaries of basic notions in our task, then we show the annotation process of the dataset. after that, we give fundamental statistics and analysis. ## baselines in this study, we consider synesthesia detection as a pipeline system: we first extract the sensory word from a sentence, we then detect the original sensory modality and specific usage of sensory modality for the sensory word. if the usage of the sensory word is still related to one sensory modality but not the original sensory modality, we consider the usage as the synesthetic sensory modality of the sensory word. therefore, we establish a family of strong and representative baselines, including sensory word extraction models presented in section 4.1, and synesthesia detection models presented in section 4.2. ## proposed method there are three challenges in synesthesia detection: 1) the sensory modality of the sensory word and its context may be different, and thus it is necessary to capture the sensory expression of the sensory word and its context. 2) the sensory word may be not a single character or word. we thus need to detect the boundary of the sensory word. 3) there is an association between original and synesthetic sensory modalities, which makes modeling interaction between them necessary. in this study, we propose a radical based neural model to address the above three challenges. as shown in figure 3 , we employ the radical-based text representation to capture the sensory expression of the sensory word and its context. we then identify the boundary of the sensory word using a machine reading comprehension model. afterward, we employ a joint learning model to detect the original and synesthetic sensory modalities collectively. ## experiments in this section, we carry out various experiments to investigate the effectiveness of the proposed method on the synesthesia detection task. in addition, we empirically compare the proposed model and the selected baselines. ## conclusion in this paper, we define a new task called chinese synesthesia detection. in particular, we construct a large-scale manually annotated chinese synesthesia dataset. based on this dataset, we establish a family of baseline models. furthermore, we propose a radical-based neural network model to identify the boundary of the sensory word, and to detect the original and synesthetic sensory modalities jointly. through extensive experiments, we verify the importance of the new task and the chinese synesthesia dataset. in addition, we observe that our proposed model yields state-of-the-art results on the synesthesia dataset.