Smart Word Suggestions for Writing Assistance (2023)

Abstract: Enhancing word usage is a desired feature for writing assistance. To further advance research in this area, this paper introduces “Smart Word Suggestions” (SWS) task and benchmark. Unlike other works, SWS emphasizes end-to-end evaluation and presents a more realistic writing assistance scenario. This task involves identifying words or phrases that require improvement and providing substitution suggestions. The benchmark includes human-labeled data for testing, a large distantly supervised dataset for training, and the framework for evaluation. The test data includes 1,000 sentences written by English learners, accompanied by over 16,000 substitution suggestions annotated by 10 native speakers. The training dataset comprises over 3.7 million sentences and 12.7 million suggestions generated through rules. Our experiments with seven baselines demonstrate that SWS is a challenging task. Based on experimental analysis, we suggest potential directions for future research on SWS. The dataset and related codes will be available for research purposes.

## introduction writing assistance is a widely used application of natural language processing (nlp) that helps millions of people. in addition to common features like grammatical error correction @xcite , paraphrasing @xcite and automatic essay scoring @xcite , providing word suggestions is a desired feature to enhance the overall quality of the writing. as illustrated in figure 1 , the word "intimate" in the first sentence should be replaced with "close", as "intimate" is not suitable for describing relationships between colleagues. in this paper, we introduce the task and benchmarks of smart word suggestion (sws). figure all samples consist of sentences annotated with multiple improvable targets, each of which is further annotated with multiple substitution suggestions. to save space, the sentences are simplified, and only one target and one suggestion are presented per case. the suggestions can be divided into two types: refine-usage and diversifyexpression, which are described in section 3.1 2 shows the definition of sws. the goal of sws is to identify potential improvable targets in the form of words or phrases within a given context, and provide substitution suggestions for every improvable target. these suggestions may include correcting improper word usage, ensuring that language usage conforms to standard written conventions, enhancing expression, and so on. specifically, we categorize these suggestions into two types: refine-usage and diversify-expression. lexical substitution (ls) @xcite @xcite is the most relevant research benchmark in the field. ls systems aim to provide substitute words that support / assistance / guide / aid ## related works we begin by comparing sws with three related tasks, highlighting the unique value of our work. ## data collection this work is to construct a smart word suggestion benchmark that accurately represents writing assistance scenarios. for evaluation, we collect sentences from english learners and use human annotations in accordance with @xcite and @xcite . for training, we compile a large-scale, distantly supervised dataset from wikipedia @xcite . ## evaluation in this section, we introduce the evaluation settings and metrics for sws, including both the end-to-end evaluation and the sub-task evaluation. for the end-to-end evaluation and the improvable target detection sub-task, we introduce precision, recall, and f 0.5 as metrics. for the substitution suggestion sub-task, we utilize accuracy to evaluate the quality of the predicted substitutions. examples of calculating the metrics can be found in appendix b. ## analysis we analyze the experimental results with two questions: (1) does the model have the capability to accurately identify words that require improvement, or does it simply make random guesses? (2) does the model have the ability to provide multiple useful suggestions for each target word? ## conclusion this paper introduces the first benchmark for smart word suggestions (sws), which involves detecting improvable targets in context and suggesting substitutions. different from the previous benchmarks, sws presents a more realistic representation of a writing assistance scenario. our experiments and analysis highlight various challenges for future research and suggest opportunities for improvement in future work. we encourage further research on building more realistic training data, designing better data augmentation strategies, and developing unsupervised or self-supervised methods for sws.