COMPILING : A Benchmark Dataset for C hinese Complexity Controllable Definition Generation (2022)

Abstract: “The definition generation task aims to generate a word’s definition within a specific context automatically. However, owing to the lack of datasets for different complexities, the definitions produced by models tend to keep the same complexity level. This paper proposes a novel task of generating definitions for a word with controllable complexity levels. Correspondingly, we introduce COMPILING, a dataset given detailed information about Chinese definitions, and each definition is labeled with its complexity levels. The COMPILING dataset includes 74,303 words and 106,882 definitions. To the best of our knowledge, it is the largest dataset of the Chinese definition generation task. We select various representative generation methods as baselines for this task and conduct evaluations, which illustrates that our dataset plays an outstanding role in assisting models in generating different complexity-level definitions. We believe that the COMPILING dataset will benefit further research in complexity controllable definition generation.”

## introduction definition generation (dg) is the task of describing the meaning that a word takes in a specific context. this task can help language learners by providing explanations for unfamiliar words. recent researches @xcite attempted to apply the task to the field of intelligent computer-assisted language learning (icall), and have made a significant progress. previous studies on dg mainly concentrate on generating different definitions for polysemous words @xcite @xcite , or generating definitions with appropriate specificity @xcite . in these studies, researchers have faced various issues, such as the high complexity problem. high complexity definitions contain words that are more difficult than the defined word, and hence are labored for language learners to read and understand. nevertheless, there have been few focuses on complexity controllable generation of definitions. a possible reason is that the complexities of definitions are not provided in currently existed datasets, which leads to the difficulty of automatic training and evaluation. actually, the problems mentioned above are especially prominent in the language environment of chinese. definitions with suitable complexity are in urgent practical needs for chinese as foreign language (cfl) learners. according to the ministry of education of china, by the end of 2020, more than 20 million foreign students are learning chinese. but as @xcite pointed out, since the difficulty of definitions is not considered, most existing dictionaries cannot meet cfl learner's requirements. besides, the existing chinese learner dictionaries contain only a small number of words. for instance, the contemporary chinese learner dictionary (ccld) only has about 6,400 words. in contrast, the modern chinese dictionary (mcd), which is designed for native speakers, has about 69,000 words. therefore, in this work, we focus on the task of generating definitions for cfl learners with appropriate complexities. at present, there are two datasets used for the chinese definition generation task, but neither of them can meet the needs of this task. the most widely used cwn dataset @xcite @xcite was built from the chinese wordnet @xcite , which is a knowledge base of sense distinction foot_0 . this dataset is limited in size with 8,221 words. @xcite constructed a dataset from the 5th edition of mcd. but it only collects disyllabic nouns and verbs, and additional annotation of formation rules is required. besides, both datasets didn't provide the complexity of definitions, which is essential information in the controllable generation. to enhance the study of this task, we propose to build a novel benchmark dataset named compiling (chinese complexity controllable definition generation). the dataset is large and of high quality, which contains 127,757 entries in total. each entry consists of a word, an example, a definition, and two complexity measurements of this definition. more specifically, we build the dataset by using two chinese dictionaries, namely the ccld and the 7th edition of mcd. the former collects fewer words, but the definitions are simpler. the latter is the opposite. by combining these two dictionaries, we obtain a large amount of definitions that vary in different complexities. in order to quantitatively measure the complexity of definitions, we refer to the graded vocabularies formulated by hsk (chinese proficiency test). hsk is set up to test the proficiency of non-native speakers. it has nine levels from easy to hard, and each level corresponds to a vocabulary. the compiling dataset contains an average level and a maximum level of each definition. we find that both dictionaries tend to use phrases rather than complete sentences as examples in some cases. for instance, the word "规模" (scale) has two example phrases in mcd (modern chinese dictionary), which are "规模宏大" (large scale) and "初具规模" (begin to take shape). we think that short phrases might be helpful for language learners to understand, but complete sentences can provide more context in the automatic definition generation. thus, we design an algorithm to expand the phrases into sentences (section 4.2). we believe that this dataset can further enhance the research on chinese complexity controllable definition generation, which could not only benefit the language learners, but also low literacy readers, as well as people with aphasia or dyslexia. we also provide baselines of mainstream generation methods as references (section 6). in summary, our contributions are listed below: notably, @xcite proposed to generate simple definitions employing a multitasking framework. since the lack of a definition dataset with different complexities, they managed to generate both complex and simple definitions in an unsupervised way. differently, we focus on building the benchmark dataset for different chinese definition generation tasks and hope it could be beneficial for further research. ## problem formulation in this work, we aim to generate a definition d c with appropriate complexity c, for a given word and example sentence (w * , e). this task is feasible because the word and it's corresponding definition should be assumed to have the same semantics. a common solution is to predict tokens in the definition one by one, depending on the previous words and the other conditions, which can be formulated as: ## dataset construction the source corpora are extracted from the mcd and ccld, both published by the commercial press. for corpus from mcd and ccld, we process them separately with the same construction methods and finally put them together. the construction of the compiling dataset is divided into three stages: data structured annotation, example sentences expansion, and post-processing. first, we propose a strategy for building structured datasets due to the high complexity and compact construction of automatically extracted data. in this phase, we set up a platform. it not only helps annotators proofread and audit corpus data more efficiently but is also conducive for us to check and collect data. besides, since the context of a targeted word in the dictionary is always a collocation instead of a complete sentence, we then conduct expanding context to enhance the overall abundance of language for our proposed datasets. furthermore, to divide definitions into different complexity levels, we calculate the hsk level of each description. ## dataset analysis as shown in hence, the compiling dataset could be applied to both general definition generation tasks and those which incorporate the complexity of definitions, demonstrating its value in being as a benchmark dataset. log-cad log-cad @xcite is a model for generating descriptions for words and phrases. this model summarizes clues from the static, contextualized, as well as character-level em-c c l 2 0 2 2 beddings of the given word, and then employs an lstm-decoder for the generation. a gated attention mechanism is employed to capture and filter information from the embeddings during decoding. transformser we treat the task as a special type of single language translation and directly use the original transformer model proposed by @xcite . we concatenate the word and example sentence as the input sequence and train the model to generate the definition. we use the same approach to deal with the input and output in bert and bart models. all hyper-parameters are set according to the original paper for a fair comparison. bert pretrained language models have been widely used in various nlp tasks in recent years. by obtaining prior knowledge during pretraining, the plms can encode the input sentence more effectively. thus, we use the chinese-bert-base @xcite model to initialize all the parameters in a transformer encoder and employ a transformer decoder for the generation. note that the decoder is trained from scratch without initialization. bart unlike bert, bart @xcite is a pretrained encoder-decoder language model, which is more suitable for generation tasks. since the monolingual bart only support english, we use the multilingual version of bart and set both source and target language as chinese for this task. mt5 t5 is one of the representative pre-training language models. it considers all nlp tasks as a uniform text-to-text paradigm. mt5 @xcite is a multi-language variant of t5, and its performance on various benchmark tasks is generally outstanding. therefore, we choose mt5 to perform the prompt learning method. ## conclusion in this work, we propose a novel task of generating chinese complexity controllable definitions for a given word and example sentence. this task is of great use in helping cfl learners and low literacy readers. meanwhile, we introduce the compiling dataset, which is a benchmark adapting to kinds of definition generation tasks. we also provide several baselines for this task, among which the prompt learning method better assist models in generating definitions with specified complexity. nevertheless, the experimental results also show that this task is challenging, and the performance needs further improvement.