C hinese Lexical Substitution: Dataset and Method (2023)

Abstract: Existing lexical substitution (LS) benchmarks were collected by asking human annotators to think of substitutes from memory, resulting in benchmarks with limited coverage and relatively small scales. To overcome this problem, we propose a novel annotation method to construct an LS dataset based on human and machine collaboration. Based on our annotation method, we construct the first Chinese LS dataset CHNLS which consists of 33,695 instances and 144,708 substitutes, covering three text genres (News, Novel, and Wikipedia). Specifically, we first combine four unsupervised LS methods as an ensemble method to generate the candidate substitutes, and then let human annotators judge these candidates or add new ones. This collaborative process combines the diversity of machine-generated substitutes with the expertise of human annotators. Experimental results that the ensemble method outperforms other LS methods. To our best knowledge, this is the first study for the Chinese LS task.

## introduction lexical substitution (ls) aims at finding appropriate substitutes for a target word in a sentence, which can be used as a backbone of various nlp applications such as writing assistance @xcite , word sense disambiguation @xcite , and lexical simplification @xcite . for instance, when presented with the sentence "i read an amazing paper today", we aim to select a more descriptive adjective to substitute the word "amazing". while options such as "awesome" and "great" may readily come to mind, it proves arduous for us to conceive of equally fitting alternatives such as "incredible" and "fascinating". despite extensive research conducted on lexical substitution (ls) in various languages, including english @xcite @xcite @xcite , german @xcite , italian @xcite , and croatian @xcite , chinese ls has received limited attention. in this paper, we address this gap by focusing on the chinese ls task. to enable the development and evaluation of effective chinese ls methods, a large-scale dataset is intuitively important. existing widely used english ls benchmarks, ls07 @xcite , coinco @xcite , and swords @xcite , were collected by asking human annotators to think of substitutes from memory. the annotation method has the following two problems. to address these challenges, we propose a novel annotation method to construct an ls dataset based on human and machine collaboration. firstly, we propose an ensemble method that leverages four different unsupervised ls methods to automatically generate substitutes. automated methods can quickly generate a vast pool of potential substitutes, reducing the burden on human annotators. secondly, we let human annotators assess the suitability of these alternatives as substitutes. additionally, we request annotators to suggest new alternatives that are not present in the machine-generated options. this collaborative process harnesses the expertise of human annotators while leveraging the efficiency and scalability of machine-generated candidates. this efficiency allows for the creation of a larger dataset within a reasonable budget. the annotation method is motivated by the following two findings: in summary, our contributions are listed below: the dataset and code is available at github foot_0 . in english, the first ls dataset from semeval 2007 (ls07) @xcite , consists of 300 development and 1,710 test instances for 201 polysemous words. for each target word, 10 sentences are provided. the annotators' task deployed by amazon mechanical turk was to give up to 3 possible substitutes. afterward, biemnann @xcite created a large-scale dataset (twsi) that annotates 25k sentences from wikipedia, which, however, only covers noun targets. to alleviate this limitation, kremer et al. @xcite proposed concept in context (coninco), a dataset of 2,474 sentences covering 3,874 distinct targets with different part-of-speech tags, which is the current largest ls benchmark. it consists of 15k target instances with a given 35% development and 65% test. recently, stanford word substitution benchmark (swords) @xcite is built on coinco by asking human annotators for higher coverage and higher quality. swords consists of 1250 instances with a given 417 development and 833 test. considering the size of vocabulary in english, the size of the vocabulary covered by ls datasets is too small. additionally, we found that many appropriate substitutes for many instances in swords are missing, since human annotators frequently utilize repetitive patterns to fabricate instances, leading to a lack of linguistic diversity @xcite . the german ls dataset from germeval 2015 consists of 2,040 instances from the german wikipedia, which contains 153 unique target words. italian ls dataset from evalita 2009 consists of 2,310 instances, which contains 231 unique target words. all the above ls datasets in all languages are constructed by human annotators. due to their relatively small size, all of these datasets can only be used for evaluation and not for training. unfortunately, research on chinese ls is still scarce: to the best of our knowledge, there is currently no publicly available ls corpora for training, even lacking a dataset to evaluate the ability of ls models. lexical substitution. ls methods can be divided into four types: (1) dictionary-based method @xcite , (2) embeddingbased method @xcite , (3) bert-based method @xcite @xcite , and (4) paraphraser-based method @xcite . the early lexical substitution studies obtain synonyms by searching linguistic resources, such as wordnet. embedding-based methods utilize word embedding modelings to obtain highly similar words as the substitutions. since 2019, ls methods based on pretrained language models have attracted much attention @xcite @xcite , in which pretrained bert is most used. zhou et al. @xcite apply dropout to the embeddings of target words for partially obscuring the word, and obtain a probability distribution over the bert output vocabulary. arefyev et al. @xcite present a comparative study of popular pretrained language models, such as elmo, bert, and xl-net. lacerra et al. @xcite ) first merge the development set of two ls datasets (co-inco and twsi), and split it into training and development sets for fine-tuning the encoder-decoder framework. @xcite propose a new mix-up embedding strategy by incorporating the knowledge of word-net into the prediction process of bert. recently, qiang et al @xcite propose a method parals that utilizes a pretrained paraphraser to generate the substitutes. compared to language modeling, paraphraser produces fluent, meaningpreserving paraphrases but contain variations in word choice. parals achieves good results and is considered the state-out-of-art ls method. ## creating chnls in this section, we describe our method to build an ls dataset, and the overall architecture for constructing the chinese ls corpus is illustrated in figure 1 . ## conclusions this study presents the first comprehensive exploration of the chinese lexical substitution (ls) task. we propose a novel annotation method to construct a large-scale chinese ls dataset through a collaborative human-machine approach. the constructed dataset consists of 33,695 instances and 165,105 substitutes with high quality and high coverage. our proposed ensemble method by leveraging the strengths of each method while mitigating their weaknesses, our ensemble approach significantly outperforms the individual ls methods across all evaluation metrics. in conclusion, our study fills the research gap on how to construct a large-scale ls dataset with high coverage and low cost, providing a solid foundation for further research and development. the construction of a high-quality dataset and the development of an effective ensemble method showcase the potential for improved lexical substitution in the chinese language.