Improving Synonym Recommendation Using Sentence Context (2021)

Abstract: Traditional synonym recommendations often include ill-suited suggestions for writer’s specific contexts. We propose a simple approach for contextual synonym recommendation by combining existing human-curated thesauri, e.g. WordNet, with pre-trained language models. We evaluate our technique by curating a set of word-sentence pairs balanced across corpora and parts of speech, then annotating each word-sentence pair with the contextually appropriate set of synonyms. We found that basic language model approaches have higher precision. Approaches leveraging sentence context have higher recall. Overall, the latter contextual approach had the highest F-score.

## introduction writers often rely on thesauri to recommend synonyms to replace a given word. though many provide usages of synonyms within example contexts, they cannot account for the specific context of the word "live", i.e., in a writer's specific context. thus many synonym recommendations are inappropriate, leading writers to reject most synonyms proffered. we propose and evaluate a simple technique to address this problem by combining large scale pre-trained language models and hand-curated thesauri to improve the precision of synonym recommendations in context. early research on synonyms and semantic analysis began in the early 1960s and 1970s @xcite and semantic network representations were popularized by quillian @xcite , collins @xcite , woods @xcite , brachman @xcite and others. simplifying these earlier theoretical concepts, the princeton wordnet was created @xcite . wordnet is a online reference system constructed of synonym sets (synsets) which utilize semantic inheritance to describe word relations. it has served as a widely-adopted baseline tool for synonym retrieval and synset creation. the nlp community has since developed language models, e.g., word2vec @xcite , bert @xcite where nearest neighbors in these models' embedding spaces are often synonyms. however, large scale language models have been recently criticized for their complexity, bias, and high energy consumption of training on large-scale web crawls @xcite . fortunately, pre-trained versions of large scale language models are conveniently available, e.g., sen-tencetransformers foot_0 . this allows researchers or practitioners to use and extend these models for a variety of applications, helping to amortize some costs incurred during training of large scale models. in addressing the challenges of synonym recommendation and sustainable nlp, this paper makes three contributions: (1) a simple nlp technique using pre-trained language models for contextual synonym recommendation; (2) an approach for producing effective evaluation datasets for this task; and (3) a set of human annotations for the above dataset for our current and future evaluations. ## methodology we identified two baseline synonym tools that rely on token-based lookups -wordnet @xcite synsets (using the nltk foot_1 package @xcite ) and pydictionary foot_2 (using results from synonym.com) -and an existing counterfactual generative model, polyjuice @xcite , to provide a baseline comparison for our novel contextual synonym models. both wordnet and pydictionary are freely available without api restrictions and commonly used nlp tools, while polyjuice is a recently released gpt-2 @xcite model fine-tuned for counterfactual generation. two of polyjuice transformations were relevant: lexical and resemantic replacement. lexical replacements suggest alternatives to single words or noun chunks, without altering the part of speech. resemantic replacements generate replacement phrases that do not alter the dependency tree. we introduce two novel semantic context wrapper models that leverage pre-trained embeddings to identify synonyms of best fit in context of use. in the basic context model, we leverage the general similarity of contextual token embeddings of suggested synonyms to original tokens. this allows us to separate general but unlikely synonyms, edge cases, or noise from the base thesaurus suggestions from those that are generally appropriate in practical use. for example, this is particularly advantageous for tokens that can be used as multiple parts of speech such as "address" or "lead". our second contextual model, sentence context, relies on the distance from the original sentence when the suggested synonym is used in replacement. both models require a base thesaurus (e.g., wordnet), and we evaluate the performance of three base thesauri: the wordnet and pydictionary baselines individually, and the union of wordnet and pydictionary suggestions. we illustrate the difference in calculating the similarity in the embedding space between the basic (or general) context and sentence context wrappers in figure 1 for an example sentence: "i'd rather finish my tea, said the hatter, with an anxious look at the queen, who was reading the list of singers". for both wrappers, we use the same pre-trained sentence-level embedding model, "msmarco-distilroberta-base-v2" foot_3 using the imple- ## evaluation we curated a balanced annotated dataset of synonyms in context using a vocabulary of candidate synonyms from existing datasets used for synonym research. our dataset includes examples of their usage in sentences taken from open source datasets covering literary, news, and technical writing. ## conclusions and future work our main finding was that sentence context-based models improve overall performance by trading off a small loss in precision for a large gain in recall. as both techniques were nearest neighbor based, the sentence context approach allowed acceptable synonyms to be identified for the specified context that were less applicable in a general context (i.e., farther away in the word embedding used by the basic context model). an interpretation of this is that pure word embedding techniques will be most effective for the most frequently used synonyms, but sentence context techniques perform better to identify synonyms in less common usage. this is also an area where there is the greatest potential impact for contextual synonym approaches over traditional thesauri or synsets. we note that overall f-scores were low, indicating the task was challenging and has room for model improvements. for example, a limitation of our approach is the reliance on hand curated synsets or thesauri, i.e., wordnet and pydictionary. the model's true recall is bounded by the recall of these thesauri. during annotation, there were several cases where none of the recommendations were appropriate, but the annotators remarked they could recall acceptable synonyms. using proprietary thesauri could be used to improve model performance, e.g., the miriam-webster api 11 provides synonyms faceted by verb case as well as contextual sentence examples. furthermore, limiting context to phrases between the entire sentence and a single word level could further improve performance by eliminating irrelevant sentence context. beyond synonym recommendation for end users, our technique could be used to perturb model input to help quantify model robustness, for counterfactual explanations, or for query rewriting.