C hengyu STS : An Intrinsic Perspective on M andarin Idiom Representation (2025)

Abstract: Chengyu, or four-character idioms, are ubiquitous in both spoken and written Chinese. Despite their importance, chengyu are often underexplored in NLP tasks, and existing evaluation frameworks are primarily based on extrinsic measures. In this paper, we introduce an intrinsic evaluation task for Chinese idiomatic understanding: idiomatic semantic textual similarity (iSTS), which evaluates how well models can capture the semantic similarity of sentences containing idioms. To this purpose, we present a curated dataset: ChengyuSTS. Our experiments show that current pre-trained sentence Transformer models generally fail to capture the idiomaticity of chengyu in a zero-shot setting. We then show results of fine-tuned models using the SimCSE contrastive learning framework, which demonstrate promising results for handling idiomatic expressions. We also presented the results of DeepSeek for reference

## introduction among multi-word expressions (mwes), idiomatic expressions (ies) are distinctive as they are often non-compositional, suggesting their meanings may not be straightforwardly derived from individual words @xcite . for example, the phrase spill the beans can be interpreted either literally as tipping over the pulse, or figuratively as revealing a secret. the mix of compositionality and noncompositionality of ies has imposed significant challenges on various natural language processing (nlp) tasks, including sentiment analysis, machine translation, and more. proper representation of ies hence holds significant impacts on linguistics and nlp research. the present study brings its attention to ies in the chinese language or 成语 (chengyu), which lit-画蛇添足 literal: figurative 1: to ruin the effects by adding something superfluous figurative 2: to fabricate something out of thin air in contrast to idioms in western languages, which usually resemble regular phrasal constructions @xcite , chinese idioms frequently deviate from the syntactic norms of modern mandarin. they are typically compact and synthetic in structure, and exhibit opacity in semantics @xcite , as shown in researchers have made efforts to improve ie representations. for instance, @xcite have created context-aware representations for english pies, by unifying bart's @xcite ability to generate compositional meanings with an idiomatic adapter that captures their noncompositional meanings @xcite . for chinese ies, @xcite have proposed to boost idiomatic representations by mitigating the inconsistency between different semantic representations using multi-semantic contrastive learning. however, evaluations of chinese idiomatic representations so far have been carried out in more limited settings, compared to english studies. idiomaticity representation can be accessed extrinsically, through performance on downstream tasks such as machine translation, or intrinsically, by probing the linguistic properties encoded within the representations @xcite . while studies on english ies leverage both extrinsic and intrinsic methods (e.g. @xcite @xcite , varying in task genres, work on chinese idioms has largely depended on clozestyle tasks (e.g. @xcite . while such tasks directly assess language ability and are intrinsic in nature, they emphasize specific contextual use and are thus relatively less intrinsic than evaluations based on sts or embedding analysis, which focus more on semantic structure and are therefore reflect a deeper level of intrinsic evaluation. for simplicity, we refer to the former as relatively extrinsic, without implying the standard extrinsic evaluation based on downstream tasks. the observed limitations arise from multiple aspects, including a lack of diverse evaluation methods and datasets. in light of this, the study takes the initiative to explore the intrinsic evaluation of idiomatic representation in mandarin. inspired by tayyar @xcite , we adopt the ists task (see section 3) as an intrinsic method and present an sts (semantic textual similarity) benchmark on chinese idioms: chengyusts. the dataset could provide a novel alternative for assessing chinese idiom representations from an intrinsic perspective. additionally, we investigated the performance of several transformer models and deepseek (deepseek-ai, 2025) on the dataset. to our knowledge, this is the first evaluation of this new llm (large language model) on chinese idiomatic representations. plus, we introduce several baseline models trained with contrastive learning, with the best ones yielding results comparable to deepseek. the following sections are organized as follows: section 2 introduces some related work and section 3 details the task setup and dataset creation. section 4 presents the evaluation setup and results. finally, section 5 concludes the current study and suggests directions for future work. ## experiments to establish baseline performance for the chengyusts dataset, we use plms to generate sentence embeddings for each sentence pair and then evaluate their alignment with annotations by measuring their pairwise cosine similarity. we first evaluated the performance of current models on the chengyusts dataset in a zero-shot setting in section 4.1, where a model was tested without continuing pre-training or fine-tuning on the chengyusts data. subsequently, we presented fine-tuned models under the contrastive learning framework in section 4. ## conclusion and future work chengyu idioms are an essential component of chinese linguistics and cultural heritage, yet their computational processing remains underexplored in nlp, due to limited task-specific datasets and evaluation frameworks. in this paper, we have introduced ists (tayyar madabushi et al., 2021), the idiom-aware se-mantic textual similarity task, into chinese, accompanied by a curated dataset chengyusts. our experiments reveal that pre-trained sentence transformer models fail to capture idiomaticity in mandarin under the zero-shot setting, and we presented fine-tuned models using simcse @xcite , which significantly improved the performance. finally, our work is the first presenting evaluation results for the deepseek model on chinese idiom representation. deepseek achieved by far the strongest performance, and it will likely represent the reference for future task approaches.