Probing Simile Knowledge from Pre-trained Language Models (2022)

Abstract: Simile interpretation (SI) and simile generation (SG) are challenging tasks for NLP because models require adequate world knowledge to produce predictions. Previous works have employed many hand-crafted resources to bring knowledge-related into models, which is time-consuming and labor-intensive. In recent years, pre-trained language models (PLMs) based approaches have become the de-facto standard in NLP since they learn generic knowledge from a large corpus. The knowledge embedded in PLMs may be useful for SI and SG tasks. Nevertheless, there are few works to explore it. In this paper, we probe simile knowledge from PLMs to solve the SI and SG tasks in the unified framework of simile triple completion for the first time. The backbone of our framework is to construct masked sentences with manual patterns and then predict the candidate words in the masked position. In this framework, we adopt a secondary training process (Adjective-Noun mask Training) with the masked language model (MLM) loss to enhance the prediction diversity of candidate words in the masked position. Moreover, pattern ensemble (PE) and pattern search (PS) are applied to improve the quality of predicted words. Finally, automatic and human evaluations demonstrate the effectiveness of our framework in both SI and SG tasks.

## introduction the simile, which is a special type of metaphor, is defined as a figurative expression in which two fundamentally different things are explicitly compared, usually using "like" or "as" @xcite . it is widely used in literature because it can inspire the reader's imagination @xcite by giving a vivid and unexpected analogy between two objects with similar attributes. a simile sentence usually contains three key elements: the tenor, the attribute and the vehicle, foot_0 which can be defined in the form of a triple (tenor, attribute, vehicle) @xcite . for example, the simile sentence "love is as thorny as rose" can be extracted as the triple (love, thorny, rose), where the tenor is "love", the vehicle is "rose", and the attribute is "thorny". note that a simile triple can produce different simile sentences with different templates. for the example triple above, the simile sentences can be also constructed as "love is thorny like rose" with the pattern "tenor is attribute like vehicle". the study of simile is benefit to many downstream tasks, like sentiment analysis @xcite , question answering @xcite , writing polishment @xcite and creative writing @xcite . simile interpretation (si) @xcite and simile generation (sg) @xcite are the two important tasks in the study of simile @xcite . the si task is to find suitable attributes as a mediator between the tenor and vehicle. likewise, the sg task is to select a proper vehicle for the tenor with the given attribution. and these two tasks can be unified into the form of simile triple completion (stc) @xcite as shown in figure 1 . previous works on the si and sg tasks relied on a limited training corpus or labor-intensive knowledge base, which leads to an upper limit on the diversity of results. @xcite collected sentences containing comparator words from a chinese essays corpus and manually annotated them to obtain the simile triple. some works @xcite @xcite relied on a knowledge base such as conceptnet foot_1 , framenet foot_2 , which are scarce to other languages because it is time-consuming and labor-intensive to construct such a knowledge base. it is notable that pre-trained language models (plms) @xcite have made significant progress recently in many nlp tasks since it learns generic knowledge such as grammar, common sense from a large corpus @xcite . considering the sufficient existence of simile in the large corpus, it's reasonable to assume that plms are equipped with rich knowledge of similes during the pre-training stage. however, few works have explored directly probing the knowledge of simile from the plms. in this paper, we propose a unified framework to solve the si and sg tasks by mining the knowledge in plms, which does not require fine-labeled training data or knowledge graphs. the backbone of our method is to construct masked sentences with manual patterns from an incomplete simile triple, and then use language models with mlm heads to predict the masked words over the task-specific vocabulary. we take the k words with the highest probability as the result words. however, there are problems with this crude approach. firstly, the predicted words should be creative and surprised for the simile sentence. on the contrary, the plms tend to predict common words (e.g., good, bad) with a higher probability. to address this issue, we introduce a secondary pre-training stage -adjective-noun mask training (ant), where only the noun or adjective contained in the amod dependencies @xcite could be masked in the mlm training process and the number of words masked times are limited. secondly, the words predicted by mlm have a preference for different patterns. for this reason, we employ a pattern ensemble to obtain high-quality and robust results. finally, we also introduce a prompt-search method to improve the quality of the simile component predictions. our main contributions are as follows: to the best of our knowledge, it is the first work to introduce pre-trained language models to unify these tasks. ## method in this section, we will introduce our proposed method of probing simile knowledge from pretrained models. our method first introduces a secondary pre-training stage -adjective-noun mask training (ant) based on pre-trained language models to acquire diverse lexical-specific words. then two modules of pattern ensemble and pattern search are used to obtain the high-quality predictions. the framework of our method is shown in figure 2 in detail foot_3 . ## conclusion and future work in this paper, from the perspective of simile triple completion, we propose a unified framework to solve the si and sg tasks by probing the knowledge of the pre-trained masked language model. the backbone of our method is to construct masked sentences with manual patterns from an incomplete simile triple, and then use language models with mlm heads to predict the masked words. moreover, a secondary pre-training stage (the adjectivenoun mask training) is applied to improve the diversity of predicted words. pattern ensemble (pe) and pattern search (ps) are further used to improve the quality of predicted words. finally, automatic and human evaluations demonstrate the effectiveness of our framework in both si and sg tasks. in future work, we will continue to study how to mine broader or complex knowledge from pretrained models, such as metaphor, common sense and we expect more researchers to perform related research.