Predicting Antonyms in Context using BERT (2021)

Abstract: We address the task of antonym prediction in a context, which is a fill-in-the-blanks problem. This task setting is unique and practical because it requires contrastiveness to the other word and naturalness as a text in filling a blank. We propose methods for fine-tuning pre-trained masked language models (BERT) for context-aware antonym prediction. The experimental results demonstrate that these methods have positive impacts on the prediction of antonyms within a context. Moreover, human evaluation reveals that more than 85% of predictions using the proposed method are acceptable as antonyms.

## introduction antonymy is a relationship between two words that express contrasting or opposite meanings (e.g., "agree-disagree"). capturing antonymy is directly helpful for downstream applications such as sentiment transfer @xcite and claim generation @xcite . further, semantically contrasting expressions with antonyms are utilized in advertising slogans @xcite , political speeches @xcite , and chinese poetry @xcite . as antonymy is one of the relations of lexical semantics, such as synonymy and hyponymy, antonymy can be modeled using a similar approach to lexical knowledge acquisition. most of the published studies on this topic have focused on the prediction of the relation between a given word pair @xcite , or a target (tail) for a given word (head) and its relation @xcite . however, predicting an antonym is challenging because multiple types of words are plausible as antonyms for a word. this is because a word can have semantic contrastiveness to the other as long as the word contains at least one feature contrasting to the other @xcite . for example, dual, double, and multiple can be antonyms for single because they all have the contrasting features of amount or number. additionally, the appropriateness of an antonym varies depending on the context. for example, double, dual, and multiple are used for a bed, nationality, and the number of meanings of a word, respectively. hence, antonym prediction must be considered within the context. in this study, we consider the new task of antonym prediction, that is, the fill-in-the-blanks problem for antonyms in context. for example, in the sentence, "a bed is better than single for me," we expect to fill the blank with the words "double" or "king-sized." the fill-in-theblanks setting requires the prediction of contextaware antonyms by capturing the contrasting features between the word pair. the task also requires a consideration of the naturalness of a text when filling the blank, which is necessary for applications of generating text with antonyms. in recent years, pre-training and fine-tuning approaches have achieved high performance in various nlp tasks @xcite . therefore, we use bidirectional encoder representations from transformers (bert) as a pre-trained model to predict antonyms in a context. however, it is not easy to collect training data for fine-tuning the model, that is, text containing antonym pairs with a contrastive context. therefore, we focus on the rhetorical device that effectively employs antonymy, that is, antithesis, which juxtaposes words or phrases in a similar structure with contrasting meanings. an antithesis is suitable for data creation because it ensures that a text has one or more antonym pairs in a contrastive context. for example, the sentence, "my mother who [is sensitive to the pension] [is insensitive to the insurance]," has an antithesis structure with two antonym pairs. we propose four methods to finetune bert for antonyms: (1) domain adaptation using an antithesis corpus, (2) contrastive masking to focus on antonym prediction, (3) antithesis po-sitional encodings to capture antithesis structures, and (4) pseudo-supervision data collected by automatic annotation using an antonym dictionary. the experimental results with the japanese slogan corpus demonstrate that the proposed finetuning methods contribute to the adaptation of bert to the context-aware antonym prediction task. an automatic evaluation based on a single correct answer is improper because there are multiple acceptable answers. however, the manual evaluation revealed that more than 85% of the words predicted by the proposed method are appropriate as antonyms and that fine-tuned bert is highly capable of capturing antonymy in a context. ## conclusion in this study, we addressed the task of predicting antonyms within a context. we proposed methods for adapting bert to antonym prediction, such as domain adaptation using an antithesis corpus, contrastive masking, antithesis positional encodings, and pseudo-supervision data collection. the proposed method achieved 29.3% top-1 and 53.8% top-10 accuracies on the test data. although these values seem low, an automatic evaluation based on a single correct word underestimates the performance because multiple valid words can fill in the blanks. the subjective evaluation revealed that more than 85% of the words predicted by the proposed method were appropriate as antonyms. our proposed task and method will be useful in many real-world applications that use contrastive expressions. although we used japanese text in this study, it can be applied to any language as far as the annotated data is available. in the future, we will extend the proposed method to generate text with antithesis, and explore the fill-in-the-blanks problem setting for other semantic relations.