Exploring Metaphoric Paraphrase Generation (2021)

Abstract: Metaphor generation is a difficult task, and has seen tremendous improvement with the advent of deep pretrained models. We focus here on the specific task of metaphoric paraphrase generation, in which we provide a literal sentence and generate a metaphoric sentence which paraphrases that input. We compare naive, “free” generation models with those that exploit forms of control over the generation process, adding additional information based on conceptual metaphor theory. We evaluate two methods for generating paired training data, which is then used to train T5 models for free and controlled generation. We use crowdsourcing to evaluate the results, showing that free models tend to generate more fluent paraphrases, while controlled models are better at generating novel metaphors. We then analyze evaluation metrics, showing that different metrics are necessary to capture different aspects of metaphoric paraphrasing. We release our data and models, as well as our annotated results in order to facilitate development of better evaluation metrics.

## introduction metaphors are ubiquitous in human language, and while humans seem capable of easily understanding even complex metaphors, it remains difficult to implement computational methods that capture the depth and breadth of meaning inherent in novel metaphors. while many approaches have been implemented for metaphor detection, interpretation, and generation, there remain many open questions about how to incorporate linguistic and cognitive theory into these methods, as well as how to evaluate them quickly and effectively. our task is that of metaphoric paraphrase generation: given a literal input sentence, we aim to generate a semantically similar sentence that is metaphoric. specifically, we focus on two possible methods for metaphoric paraphrase generation: free and controlled. in free generation, models are trained using literal and metaphoric pairs, but are not given information regarding which metaphoric meaning is intended. this allows the model the freedom to develop new metaphors that can be creative, novel, and perhaps even inspiring. however, it can lead to noisy generation, which is difficult to evaluate as even randomly generated metaphors can sometimes be effective via coercion-like processes. additionally, if a generated text is to remain coherent, metaphor generation must be consistent within it, and free generation is thus ill-suited to generating longer coherent metaphoric texts. the other option, controlled generation, involves adding additional constraints the generation objective to encourage the model to generate a specific metaphor. this methodology is useful when an intended metaphor is known, or needs to be consistent across a dialogue: the model can be suitably constrained to be consistent. recent work has explored controlled generation @xcite ; we aim to address if and how adding control improves over free generation. as a theoretical basis for metaphors, we employ conceptual metaphor theory (cmt). in cmt, we consider linguistic metaphors as arising from conceptual metaphors that are a part of our cognitive processes, in which a concrete source domain is used to better understand a more abstract target domain @xcite . we typically consider language from the abstract target domain in the context sentence as literal, and aim to generate expressions evoking a concrete source domain that can be used to describe it metaphorically. in free generation, the system is left open to choose which domain is appropriate; in controlled generation, we provide the model explicitly with the required domains. consider the following example of free generation. we train a seq2seq metaphor model to take a literal sentence as the context and produce some metaphoric hypothesis: ## data a key bottleneck in metaphoric paraphrasing is the lack of high quality literal/metaphoric pairs for both training and evaluation. while two datasets of hand-crafted metaphoric paraphrases are available @xcite , they both contain less than 200 instances, and are thus too small for model training and/or fine-tuning. we explore two possible datasets for this purpose: a previously used dataset which contains source and target domains via framenet frames @xcite , and a new semisupervised dataset which contains source/target information based on the metaphor-based lexical resource metanet. ## human evaluation we develop both free and controlled models for the two datasets outlined in section 3. we evaluate them on a hand-crafted test set which includes samples both within the data set they were trained on and from the unseen dataset. ## automatic evaluation evaluation of metaphor generation is a difficult task, as traditional metrics for machine translation and other tasks aim to enforce lexical and semantic similarity between the input and output sequences. as these metrics often rely on word overlap, valid metaphoric paraphrases may be punished for being overly creative. we thus implement a variety of standard evaluation metrics and evaluate their correlation with our gold standard annotations. bleu, meteor, rouge word and phrase overlap metrics from machine translation such as bleu @xcite , meteor (banerjee and lavie, 2005), and rouge @xcite are often used to evaluate paraphrasing, despite noted weaknesses @xcite . we include them here to highlight their performance on creative language, and to compare them against human evaluations. translation error rate ter is another commonly used metric in machine translation, measuring the amount of correction that is necessary for a generated output to be valid @xcite . sentbert sentbert (reimers and gurevych, 2019) provides sentence transformers to generate sentential vectors. these can be then compared using cosine distance to find semantic similarity. sentbert has proven effective for a wide variety of similarity tasks, and should also be effective at determining paraphrase quality between literal and metaphoric sentences. moverscore moverscore @xcite is a metric that uses bert and earth mover distance to measure similarity between two sentences. it uses contextual embeddings similar to sentbert, and has the potential to better represent sense-specific meanings like those associated with metaphoricity. perplexity transformer-based language mod-els such as the gpt family @xcite are extremely effective at producing text. the perplexity of a given sentence under the language model can be a proxy for sentence fluency. this approach promotes common words over rare, perhaps penalizing creativity, but can used as an evaluation metrics for generation systems @xcite . abstractness the notions of abstractness and concreteness have long been staples in metaphor detection systems @xcite . we here use the abstract/concreteness ratings from @xcite , which are based on word2vec embeddings @xcite . we evaluate the mean abstractness score, as well as the standard deviation, under the assumption that one of the main aspects of metaphoricity is the difference between abstractness levels of different conceptual domains within a sentence. ## conclusions and future work we show that adding control to the metaphor generation process by means of conceptual domains improves the metaphoricity of generated paraphrases, but can decrease fluency and semantic similarity. free generation systems are entirely viable, and may present advantages depending on the required task. second, we explore automatic evaluation metrics, showing that while previously employed metrics are capable of capturing some aspects of generation, there may be conflict between components, making multiple metrics necessary in order to best reflect the overall quality of generated outputs. these results provide multiple ways forward for metaphor generation. with regard to free generation, we see that models are strong with regard to fluency and sentence similarity: future work thus requires a stronger signal for metaphoricity. this could be achieved by using improved datasets, or metaphor-specific generation models. for controlled generation, we explore only using domains from lexical resources as proxies for conceptual domains in cmt. other theories of metaphor (including conceptual blends @xcite , class-inclusion theory @xcite , and structure-mapping @xcite ) have yet to be explored for metaphor generation. with regard to evaluation, we've shown the strengths and weaknesses of some metrics, but none perform exceptionally well at this task overall. new metrics are needed; a metric that can harmonize syntactic fluency and the diversity of metaphoric expressions is necessary for rapid model development and evaluation. sander wubben, antal van den bosch, and emiel krahmer. 2010. paraphrase generation as monolingual translation: data and evaluation. in proceedings of the 6th international natural language generation conference. qiongkai xu, juyan zhang, lizhen qu, lexing xie, and richard nock. 2018. d-page: diverse paraphrase generation. arxiv, 1808.04364. qian yang, zhouyuan huo, dinghan shen, yong cheng, wenlin wang, guoyin wang, and lawrence carin. 2019. an end-to-end generative architecture for paraphrase generation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3132-3142, hong kong, china. association for computational linguistics. zhiwei yu and xiaojun wan. 2019. how to avoid sentences spelling boring? towards a neural approach to unsupervised metaphor generation. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 861-871, minneapolis, minnesota, usa. association for computational linguistics. wei zhao, maxime peyrard, fei liu, yang gao, christian m. meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 563-578, hong kong, china. association for computational linguistics.