P ro L ex: A Benchmark for Language Proficiency-oriented Lexical Substitution (2024)

Abstract: Lexical Substitution discovers appropriate substitutes for a given target word in a context sentence. However, the task fails to consider substitutes that are of equal or higher proficiency than the target, an aspect that could be beneficial for language learners looking to improve their writing. To bridge this gap, we propose a new task — language proficiency-oriented lexical substitution. We also introduce ProLex, a novel benchmark designed to assess systems’ ability to generate not only appropriate substitutes but also substitutes that demonstrate better language proficiency. Besides the benchmark, we propose models that can automatically perform the new task. We show that our best model, a Llama2-13B model fine-tuned with task-specific synthetic data, outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex.

## introduction nowadays, automatic english learning tools have become widespread across various educational settings. for instance, automatic grammar error correction systems @xcite @xcite , simplify grammar correction for learners and help enhance their writing skills. besides grammar, enhancing vocabulary diversity is another crucial element in improving english writing @xcite @xcite . nevertheless, english second-language (l2) learners frequently struggle to use a diverse vocabulary in their writing @xcite @xcite . they tend to use the same set of words repeatedly, which may negatively impact their performance on essay writing tests @xcite . the beginner l2 learners can leverage existing lexical substitution systems @xcite @xcite @xcite to enhance their vocabulary breadth. these systems are designed to identify contextually suitable substitutes for a target word, thereby assisting learners in discovering appropriate substitutions. however, merely knowing these contextually appropriate substitutes is insufficient for l2 learners. prior benchmarks @xcite @xcite for lexical substitution focus solely on the contextual appropriateness of generated substitutes. to enhance the vocabulary diversity and writing proficiency of english learners, guided by the principles of the zone of proximal development @xcite , we propose that the substitutions should also reflect an equal or higher level of language proficiency compared to the original target word. in this work, we present prolex, a novel lexical substitution benchmark that evaluates system performances on language proficiency-oriented lexical substitution, a new task that proposes substitutes that are not only contextually suitable but also demonstrate advanced-level proficiency. to construct prolex, we begin by selecting target words and sentences according to their frequency in toefl-11 @xcite , a comprehensive corpus of essays written by non-native english learners. this approach ensures a data distribution that is more representative of l2 english learners. subsequently, to effectively harness human expertise in annotating advanced proficiencyoriented lexical substitutions, we adopt the methodology recommended by @xcite -directly having humans judge the appropriateness of candidate substitutes instead of generating them ad hoc. to facilitate this, we employ gpt-4 to ## related work we first outline the task of lexical substitution, then introduce recent high-quality benchmark in this field, swords, as detailed in @xcite . next, we describe toefl-11 @xcite , a large-scale corpus of written essays composed by non-native english learners, which we adopt as the corpus for selecting context sentences and target words for prolex. ## prolex benchmark we propose prolex, a benchmark for language proficiency-oriented lexical substitution. prolex is composed of quadruplets, each containing a target word, a context sentence, a list of acceptable substitutes, and a list of proficiency-oriented substitutes (w, s, w a , w a p ). as indicated by @xcite , previous benchmarks @xcite gathered substitute words by prompting humans to generate them ad hoc. however, many viable substitutes, challenging for humans to devise, might ultimately be overlooked. therefore, we follow @xcite by first obtaining a set of candidate substitutes and later asking annotators to judge their appropriateness. formally, given a context s, target word w, and candidate substitute w ′ , the annotators are asked to judge whether w ′ is an acceptable substitute for the target word: (w, s, w ′ ) → {1, -1, 0}, with 1, -1 and 0 indicating "accept", "reject", and "uncertain" respectively. figure 1 shows the complete benchmark creation process for prolex. in the following sections, we will first illustrate the data creation process in section 3.1. then, in section 3.2, we will describe our language-learningoriented annotation scheme. in section 3.3, we show how we filter substitutes based on cefr proficiency levels. ## dataset annotation process and statistics considering the necessity for substantial semantic understanding and knowledge of english collocations in the annotation process, we recruited two annotators currently pursuing phds in linguistics, with specializations in english language teaching and education. on the 125 (w, s) pairs, namely 625 (w, s, w ′ ) triples, the two annotators reached an inter-annotator agreement of κ = 0.60, indicating a near-substantial level of agreement. 5 then, they annotated the complete test set separately, with each annotating 2, 500 (w, s, w ′ ) triples sampled from section 3.1. in total, among all 5, 000 candidate substitutes w ′ , 39% is labeled as 1 (accept), while 51% and 10% are annotated as -1 (reject) and 0 (uncertain), respectively. this entails that gpt-4 generated candidates may not always be valid in prolex, indicating future research opportunities in using llms for generating semantically correct, collocationally appropriate english lexical substitutions. figure 3 : distribution of cefr levels for substitutes in w a and substitutes in w a p in prolex. in low-level sentences, more than 65% of the proficiency-oriented substitutes are from b1 level or higher; similarly, in medium-level sentences, over 75% of these substitutes are sourced from b1 level or above. we further process the annotations to compose the final quadruplets (w, s, w a , w a p ) in prolex, where w a denotes a list of acceptable substitutes for w, and w a p denotes the list of acceptable substitutes after filtering based on cefr levels. we exclude the target words that have zero acceptable substitutes. ## language proficiency-oriented lexical substitution in this section, we outline recommended evaluation practices for prolex, propose models that can automatically perform the task on prolex, and assess 6 the part-of-speech distribution for the target word w in prolex is shown in ## conclusion and future work we propose a new task language proficiencyoriented lexical substitution to improve the vocabulary diversity and writing proficiency of l2 english learners. we introduce prolex, a novel benchmark designed to assess systems' ability to generate appropriate and language proficiency-oriented substitutes for given target words in the context. besides, we also fine-tuned open-source language models on task-specific synthetic data, achieving results that are on par with, or better than gpt-4 and chatgpt. in future, we will expand prolex by leveraging larger l2 english writing corpora and incorporating more comprehensive sets of substitutes.