Learning to Substitute Words with Model-based Score Ranking (2025)

Abstract: Smart word substitution aims to enhance sentence quality by improving word choices, however current benchmarks rely on human-labeled data , which suffers from subjectivity and lacks diversity due to limitations in the number of annotators. Since word choices are inherently subjective, ground-truth word substitutions generated by a small group of annotators are often incomplete and likely not generalizable. To circumvent this issue, we instead employ a model-based scoring (BARTScore) to quantify sentence quality, thus forgoing the need for human annotations. Specifically, we use this score to define a distribution for each word substitution, allowing one to test whether a substitution is statistically superior relative to others. Further, we propose a loss function that directly optimizes the alignment between model predictions and sentence scores, while also enhancing the overall quality score of a substitution. Crucially, model learning no longer requires human labels, thus avoiding the cost of annotation while maintaining the quality of the text modified with substitutions. Experimental results show that the proposed approach outperforms both masked language models (BERT, BART) and large language models (GPT-4, LLaMA).

## introduction in the current era of ai-driven technologies, the generation of natural language by machines has become a critical area of research within the field of natural language processing (nlp). the advent of advanced language models such as gpt @xcite and llama @xcite ) has brought about unprecedented improvements in generating coherent, contextually accurate text across a wide range of applications, including machine translation, summarization, and conversational agents. these models are revolutionizing industries by enabling machines to produce human-like text at scale, greatly expanding the possibilities for automation and interaction in both professional and creative domains. as language generation capabilities improve, the need to refine and control the output of these models becomes more pressing. one key area in this direction is smart word suggestion (sws) defined by @xcite as a task that focuses on enhancing writing by improving two aspects of nlp: identifying words (or tokens more generally) that if replaced can improve sentence quality, and suggesting alternatives for such words. complementarily, @xcite introduced a dataset to benchmark models built for sws. this task is essential in applications such as paraphrasing, machine translation, and writing assistance, where accurate word choice directly impacts the clarity and quality of the generated text. despite the progress in fulltext generation, word substitution presents unique challenges in maintaining grammatical correctness, contextual appropriateness, and semantic fidelity. previous sws work can be divided into taskspecific models and prompt-based methods. taskspecific models are trained directly to address the sws task. one approach adapts models for lexical substitution (ls) @xcite , allowing them to generate substitutions for each word in a sentence. if the substitution differs from the original word, it is recognized and recorded as such. bert-based @xcite and the lexsubcon @xcite models were first explored in @xcite . they also fine-tuned pretrained models like bert @xcite and bart @xcite , allowing them to handle sws end-to-end. importantly, these models were fine-tuned using supervised learning to provide suggestions for wikipedia text using ground-truth substitutions obtained from a list of synonyms gathered from a thesaurus. orthogonal to these machine learning techniques, rule-based methods that rely on paraphras-ing databases (e.g., ppdb @xcite ) or a thesaurus (e.g., the merriam-webster thesaurus) have also been explored @xcite . alternatively, prompt-based techniques leveraging large language models (llms), such as gpt @xcite , have gained popularity in sws. provided that research in language generation has shifted from narrow tasks like substitution to generating coherent, contextually rich text, tasks such as sws have become less of a priority. however, in general it is still key to enhance sentence quality, as precise word choices can significantly impact clarity, tone, and overall effectiveness in communication. moreover, the success of these methods is highly dependent on prompt design. from a different perspective, benchmarks for sws predominantly rely on human-labeled data for both training and evaluating model performance. however, only a small group of annotators are recruited for data labeling, which can lead to a lack of diversity. for instance, in the sws test dataset @xcite , 12,883 suggestions for target words are available of which 7,827 (60.8%) were created by a single annotator, only 1632 (12.67%) of suggestions were created by at least three annotators, and merely 633 (4.91%) of suggestions were created by more than three annotators. while scaling up the number of annotators could potentially mitigate this issue, it significantly increases the costs of generating the annotations. in the context of substitution tasks, a large number of annotators is necessary to obtain a comprehensive representation of the substitution space, as evaluating them requires a diverse sample of high-quality alternatives. as a result, validation sets in such benchmarks may not fully capture model performance. conversely, recent research has shown that model-based evaluation metrics align closely with human judgments @xcite @xcite , offering a viable alternative to alleviate the drawbacks of human evaluations. to address these challenges, we propose an approach that eliminates the dependency on humanlabeled data. our contributions are listed below: ## related work enhancing word usage is a key feature of writing assistance. the sws task in @xcite addresses it by emulating a real-world end-to-end writing scenario. unlike traditional ls tasks @xcite , sws requires systems to identify words that, if replaced, can improve sentence quality and suggest alternatives for such words. in the ls task, however, the word to be replaced is predefined and the model is only tasked with finding suitable substitutions without altering the overall meaning of the sentence. further, in ls substitutions are typically lemmatized to match ground-truth labels, but in sws they must be in the correct grammatical form. @xcite proposed a ls method using contextual word embeddings, modifying bert with a dropout embedding policy to partially mask target words and introduced a validation score based on the top four layers of bert for candidate evaluation. @xcite integrated external knowledge from wordnet into bert for ls, combining scores from bert, wordnet gloss similarity, and sentence embeddings, along with the validation score from @xcite , to generate substitutes. parals @xcite generated substitutes through a paraphrasing model, using a heuristics-based decoding strategy. to adapt the substitution capabilities from ls to the sws task, @xcite used an approach that allowed models to generate substitutions for each word. if the substitution differs from the original word, it is recognized as the word the model wants to improve and recorded as a suggestion. further, they fine-tuned pretrained models like bert @xcite and bart @xcite to address the task in an end-to-end manner, and as an alternative to machine learning, rule-based methods using dictionaries (thesaurus) were also explored. recently, prompt-based methods have emerged as a powerful alternative for sws @xcite , by leveraging the capabilities of llms such as gpt @xcite . these approaches shift the focus from traditional model fine-tuning to designing prompts that guide the model to produce desired outcomes. while this flexibility allows for easier adaptation to various tasks, the effectiveness of the method heavily relies on the prompt's structure. despite the growing interest in generating fluent, contextually cohesive text with llms, prompt-based methods often overlook the importance of improving specific word choices. in contexts where sentence clarity and precision are key, refining word choices remains an important task that prompt-based approaches need to address. compared to traditional methods that rely heavily on human involvement, e.g., via manual annotations, in our work, we utilize model-based sentence scoring, specifically, bartscore to quantify the quality of sentences, thereby eliminating the human-associated costs. we align the model's output with the scores through a ranking loss, ensuring that substitutions with higher scores are more likely to be generated by the model, resulting in higher-quality substitutions. this approach forgoes the need for human-dependent supervised training, thus without the need for human intervention. ## methodology in sws, achieving a representative sample of highquality alternatives requires a broad pool of annotators, as diverse substitution options are essential for accurate evaluation. however, relying solely on human annotators is thus not only costly but also inefficient. below we introduce a method to train and evaluate token substitution models that does not require human annotations, by instead solely relying on model-based scoring functions. problem definition let x denote a sentence composed of n tokens, i.e., @xmath0, . . . , x n ). we aim to find a collection of k potential substitutes { wk } k @xmath1, while estimating their likelihood given the sentence (i.e., its context) as p θ (x @xmath2, where x is the original sentence and p θ (•) is a model for the conditional likelihood for token x n parameterized by θ. note that technically, if p θ (•) is a masked language model we should write p θ (x @xmath3, where x \xn indicates that x n has been masked out in x, and alternatively, if we use an (autoregressive) language model we should write p θ (x @xmath4, z), where @xmath5. however, we use p θ (x @xmath6. based on these likelihood estimates, the token substitution rule is set as x n ← wk if p θ (x @xmath7, where w n is the original value of x n . moreover, if multiple replacement candidates satisfy the substitution rule, we select wk according to argmax k p θ (x @xmath8, and alternatively, if none satisfy it, we leave the token unchanged, i.e., x n ← w n . importantly, p θ (x @xmath9, rather than the quality of the sentence. so motivated, we also seek to align such estimates with a score m (x @xmath10. for simplicity, in the following we denote the sentence x with x n replaced with the k-th candidate, wk , as @xmath11, . . . , xn ). further, to make our objective scalable and general, the score m (x @xmath12, but via a black-box model with which we can score sentences, but through which we cannot learn, i.e., gradients cannot be propagated through it to obtain learning signals. below we start by showing how we can use a model-based score to statistically characterize the suitability of a candidate as an alternative to the substitution rule introduced above, and then present an approach to estimate p (x @xmath13. ## experiments below we illustrate the problem with evaluating with human annotations, then we present an ablation study comparing the optimization approaches in section 3.2 and a benchmark comparing the proposed model to mlm and llm approaches in terms of model-based score alignment, average score and the statistic in (2). further, we explore the performance of top-2 predictions, i.e., when the top prediction is the original token, and results comparing llms with and without prompts encouraging token substitutes to be ranked by quality. ## discussion this paper addressed the problem of training and evaluating sws models without requiring human annotations. specifically, we leveraged a modelbased score (bartscore) to define evaluation metrics and to serve as a label-proxy for our loss function. extensive experimental results demonstrate the effectiveness of our approach relative to both mlm-and llm-based models.