Optimizing C hinese Lexical Simplification Across Word Types: A Hybrid Approach (2024)

Abstract: This paper addresses the task of Chinese Lexical Simplification (CLS). A key challenge in CLS is the scarcity of data resources. We begin by evaluating the performance of various language models at different scales in unsupervised and few-shot settings, finding that their effectiveness is sensitive to word types. Expensive large language models (LLMs), such as GPT-4, outperform small models in simplifying complex content words and Chinese idioms from the dictionary.To take advantage of this, we propose an automatic knowledge distillation framework called PivotKD for generating training data to fine-tune small models.In addition, all models face difficulties with out-of-dictionary (OOD) words such as internet slang.To address this, we implement a retrieval-based interpretation augmentation (RIA) strategy, injecting word interpretations from external resources into the context.Experimental results demonstrate that fine-tuned small models outperform GPT-4 in simplifying complex content words and Chinese idioms. Additionally, the RIA strategy enhances the performance of most models, particularly in handling OOD words. Our findings suggest that a hybrid approach could optimize CLS performance while managing inference costs. This would involve configuring choices such as model scale, linguistic resources, and the use of RIA based on specific word types to strike an ideal balance.

## introduction lexical simplification (ls) is the task of replacing complex words in a sentence with simpler alternatives while preserving the original meaning and structure. ls improves text readability, benefiting a wide range of people, such as students @xcite , nonnative speakers @xcite , and individuals with cognitive impairments @xcite . however, ls is a challenging task that requires both linguistic knowledge and contextual awareness. ## related work lexical simplification primarily follows a pipeline consisting of three main stages: the identification of complex words, the generation of candidate substitutes, and the selection and ranking of candidates. the identification of complex words aims to determine which words are considered complex in a sentence by a specific target population @xcite @xcite . aligning with current baseline methods, we do not focus on this stage, since complex words are given in the used dataset. knowledge-based methods early research on lexical simplification relied on lexical knowledge databases to generate substitutes @xcite . however, databases are not only expensive to develop and maintain, but also limited in word coverage. word embedding-based methods with the advent of deep learning, semantic similarity computation based on word embeddings has become a popular method for substitute generation and ranking @xcite . the cost of training word embedding models is lower than that of constructing knowledge databases, and word embeddings also alleviate the problem of word coverage. plm-based methods subsequently, pre-trained language models show strong ability in capturing contextual semantic information, and have been used for lexical simplification. for example, bert-ls @xcite introduced an unsupervised method that employs bert to generate substitutions for complex words based on the encoding of surrounding context. promptls @xcite found that fine-tuning plms can achieve better performance compared to unsupervised approaches. @xcite fine-tuned an encoder-decoder model for substitute generation, which naturally predicts simple words with multiple tokens. one major challenge in fine-tuning is the limited availability of supervised training data for certain languages, like chinese. @xcite proposed a prompt tuningbased method to alleviate this. ## task, data and preliminary analysis in this section, we provide a brief introduction to task formulation, dataset, and evaluation metrics, followed by an analysis of representative baselines that use bert and llms. ## the proposed method we propose a framework that consists of three key modules: automatic knowledge distillation, retrieval-based interpretation augmentation, and a word type-aware controller. ## conclusion this paper presents a word-type aware approach for chinese lexical simplification. we find that in unsupervised and few-shot settings, gpt-4 can obtain good performance in simplifying complex words from the dictionary, largely outperforming the small models, but still struggle to tackle ood words. considering that gpt-4 is costly, we propose an automatic knowledge distillation framework, pivotkd, to generate training data using gpt-4 for fine-tuning small models, which can outperform gpt-4 in simplifying complex words from the dictionary. for addressing the issue of ood words, we propose a retrieval-based interpretation augmentation strategy, which effectively improves the performance on ood words. consequently, we are able to configure the choice in terms of model scale, the use of linguistic resources and the interpretation augmentation strategy according to the type of complex words to strike an optimal balance between performance and cost.