Can Pre-trained Language Models Interpret Similes as Smart as Human? (2022)

Abstract: Simile interpretation is a crucial task in natural language processing. Nowadays, pre-trained language models (PLMs) have achieved state-of-the-art performance on many tasks. However, it remains under-explored whether PLMs can interpret similes or not. In this paper, we investigate the ability of PLMs in simile interpretation by designing a novel task named Simile Property Probing, i.e., to let the PLMs infer the shared properties of similes. We construct our simile property probing datasets from both general textual corpora and human-designed questions, containing 1,633 examples covering seven main categories. Our empirical study based on the constructed datasets shows that PLMs can infer similes’ shared properties while still underperforming humans. To bridge the gap with human performance, we additionally design a knowledge-enhanced training objective by incorporating the simile knowledge into PLMs via knowledge embedding methods. Our method results in a gain of 8.58% in the probing task and 1.37% in the downstream task of sentiment classification. The datasets and code are publicly available at https://github.com/Abbey4799/PLMs-Interpret-Simile .

## introduction a simile is a figure of speech comparing two fundamentally different entities via shared properties @xcite . there are two types of similes as illustrated in figure 1 , closed similes explicitly reveal the shared properties between the topic entity and the vehicle entity, such as the property "slow" shared by "lady" and "snail" in the sentence "the old lady walks as slow as a snail"; while open similes do not state the shared property such as the sentence "the old lady walks like a snail". similes play a vital role in human expression to make literal * equal contribution † corresponding author utterances more vivid and graspable and are widely used in the corpus of various domains @xcite @xcite . it is estimated that over 30% of the comparisons can be regarded as similes in product reviews @xcite . simile interpretation is a crucial task in natural language processing @xcite @xcite , which can assist several downstream tasks such as understanding more sophisticated figurative language (veale and hao, 2007) and sentiment analysis @xcite . take the simile "the lawyer is like a shark" for an example. although all words in this simile are neutral, this simile expresses a negative affect since "lawyer" and "shark" share the negative property "aggressive". in the past few years, large pre-trained language models (plms) have achieved state-of-the-art performance on many natural language processing tasks @xcite . recent studies suggest that plms have possessed various kinds of knowledge into contextual representations @xcite @xcite . however, the ability of plms to interpret similes remains under-explored. although some recent work @xcite studies the ability of plms in choosing or generating the plausible continuations in narratives, this way cannot fully reveal the ability of plms to interpret similes. in this paper, we propose to investigate the abil- " indicates the correct answer. the italicized words one by one in each sentence are the topic, masked property, and vehicle, respectively. based on our designed task, we evaluate the ability of bert @xcite and roberta @xcite to infer the shared properties of similes. we perform an empirical evaluation in two settings: (1) zero-shot, where the models are off-the-shelf; (2) fine-tuned, where the models are fine-tuned with mlm objective via masking properties. we observe that plms have been able to infer properties of similes in the pre-training stage and the ability can be further enhanced by fine-tuning. however, fine-tuned plms still perform worse than humans. moreover, we find that the simile components vehicle and topic contribute the most when inferring the properties. inspired by the sufficient hints offered by the components vehicle and topic in our empirical study, we propose a knowledge-enhanced training objective to further bridge the gap with human performance. considering property (p) as the relation between topic (t) and vehicle (v), we design a simile knowledge embedding objective function following conventional knowledge embedding methods @xcite to incorporate the simile knowledge (t,p,v) into plms. to integrate simile knowledge and language understanding into plms, we jointly optimize the knowledge embedding objective and the mlm objective in our design. overall, the knowledge-enhanced objective shows effectiveness in our probing task and the downstream task of sentiment classification. to summarize, our contributions are three-fold: (1) to our best knowledge, we are the first to systematically evaluate the ability of plms in interpreting similes via a proposed novel simile property probing task. (2) we construct simile property probing datasets from both general textual corpora and human-designed questions, and the probing datasets contain 1,633 examples covering seven main categories of similes. (3) we also propose a novel knowledge-enhanced training objective by complementing the mlm objective with the knowledge embedding objective. this method gains 8.58% in the probing task and 1.37% in the downstream task of sentiment classification. ## preliminaries on simile a sentence of simile generally consists of five major components @xcite , where four are necessary and the remaining one is optional. the four explicit components are as follows: (1) topic (or tenor): the subject of the comparison acting as source domain; (2) vehicle: the object of the comparison acting as target domain; (3) event: the predicate indicating act or state; (4) comparator: the trigger word of a simile such as as or like. the optional component property reveals the shared characteristics between the topic and the vehicle. there are two types of similes depending on whether the property is explicit or implicit @xcite . the similes which mention the property directly are named as the closed similes, while the others are open similes, as shown in figure 1 . ## empirical study on plms in this section, we first conduct a set of experiments to probe the ability of plms to infer properties in similes and then evaluate the influence of each component on the model performance. ## conclusion in this work, we are the first to investigate the ability of plms in simile interpretation via a proposed novel simile property probing task. we construct two multi-choice probing datasets covering two data sources. by conducting a series of empirical experiments, we prove that plms exhibit the ability to infer simile properties in the pre-training stage and further induce more related knowledge during the fine-tuning stage, but there is still a gap between plms and humans in this task. furthermore, we propose a knowledge-enhanced training objective to bridge the gap, which shows effectiveness in the probing task and the downstream task of sentiment classification. in future work, we are interested in exploring the interpretation of more sophisticated figurative language, such as metaphor or analogy.