M eta P ro 2.0: Computational Metaphor Processing on the Effectiveness of Anomalous Language Modeling (2024)

Abstract: Metaphor interpretation is a difficult task in natural language understanding. The development of relevant techniques in this domain is slow, mostly because of the lack of large annotated datasets and effective pre-trained language models (PLMs) for metaphor learning. Thus, we propose a large annotated dataset and a PLM for the metaphor interpretation task. Our foundation model is based on a novel anomalous language modeling (ALM) method, which we benchmark with comparable PLM baselines on the new dataset, finding that it largely improves model performance on metaphor identification and interpretation.

## introduction a metaphor is defined as using one (a single-word expression, swe) or several words (a multi-word expression, mwe) to represent a different meaning, rather than its literal meaning @xcite . interpreting the meanings of metaphors in contexts is particularly challenging for machines @xcite . there are two important tasks in linguistic metaphor processing, i.e., metaphor identification and interpretation. the former task is normally defined as a classification task, detecting metaphors on a sentence level @xcite , a word-pair level @xcite , or a token level @xcite . the latter task is normally defined as a property extraction task @xcite , a paraphrasing task @xcite , or an explanation pairing task @xcite . token-level metaphor identification, and paraphrasing-based metaphor interpretation are more supportive for natural language processing (nlp) downstream tasks, because metaphors can be paraphrased into their literal counterparts without breaking the coherence and general syntax of the original sentences, e.g., "i don't buy 1 your story" vs "i don't believe your story". 1 italics denote metaphors. machines can directly use paraphrased sequences to improve metaphor understanding in downstream tasks. @xcite have developed the largest token-level metaphor identification dataset, vu amsterdam metaphor corpus (vmc). many previous metaphor identification models were developed upon it @xcite @xcite , advancing the task significantly. in contrast, the progress of metaphor interpretation has been falling behind. this is likely because of the lack of a large paraphrasing-based metaphor interpretation dataset and an effective pre-trained language model (plm) for the learning of metaphor interpretation. we are motivated to develop a metaphor interpretation-orientated dataset, vu amsterdam metaphor corpus with paraphrases (vmc-p) foot_0 , and a new plm with a novel anomalous language modeling (alm) foot_1 pre-training paradigm. we aim to develop a dataset for training end-toend (e2e) metaphor interpretation systems. we believe that a good metaphor paraphrase dataset should contain sufficient reliable training data; metaphors in the dataset should be used in a manner similar to everyday language. compared to the latest metaphor interpretation dataset (im-pli) @xcite (to the best of our knowledge, it may be also the largest) which contains 920 manually paraphrased metaphors and idioms and 17,027 automatically paired paraphrases, our vmc-p dataset has more manually annotated metaphor paraphrases, accounting for 11,880 units, covering both swes and mwes. besides, each sentence in impli only has a paraphrase for a single target metaphor, which is dissimilar to the real-world distribution. in contrast, the sentences in our dataset can have multiple metaphors and corresponding paraphrases (2.2 metaphor units per metaphorical sentence on average). in real-world texts, it is common for a sentence to have multiple metaphors (see the example in figure 4 ). thus, our dataset is closer to real-world scenarios. there is another metaphor paraphrase dataset @xcite , while it just contains 200 gold paraphrases. our proposed pretraining paradigm, alm, is metaphor processingtailored and linguistics-informed, compared to current masked word prediction (mwp)-based plms, e.g., bert @xcite and roberta @xcite , language modeling-based plms, e.g., gpts @xcite @xcite , and sequence-to-sequence-based plms, e.g., t5 @xcite . alm is motivated by the fact that metaphors do not take their literal meanings in contexts. we hypothesize that the literal meanings of metaphoric words are likely contextually anomalous. for example, given "my car drinks gasoline", literally, neither a car can drink, nor can gasoline be drinkable. @xcite explained this as selectional preference violation (spv) -metaphors violate the selectional preference of their contexts. we hypothesize that a randomly replaced word also likely violates the selectional preference of its context. thus, we can develop a pre-training corpus with randomly replaced words to simulate the spv of metaphors. alm aims to detect anomalous replacements and retrieve original words to simulate metaphor identification and interpretation tasks, respectively. the pre-training corpus is free of human annotation as we can automatically replace an swe/mwe with a random one, achieving a large pre-training corpus with many anomalies. we use contrastive learning to simulate a human metaphor identification procedure (mip) (see appendix b, @xcite : a metaphor is identified based on the semantic contrast between its contextual and basic meanings. contrastive learning of positive and hard negative samples also helps alm to learn the boundary between them, which is an important human knowledge acquisition process @xcite . we re-train roberta-large with pre-processed wikipedia that contains hard anomalous swes and mwes. we compare the re-trained model, also termed alm, with parameter size-comparable plms, e.g., large bert, roberta, medium gpt2, and t5-base, finding that alm exceeds the best plm by 3.20% and 3.60% averaged f1 in metaphor identification and e2e interpretation tasks, based on vmc-p dataset. the fine-tuned alm for metaphor processing is termed metapro 2.0. we hope the preliminary success of our pretraining paradigm in a small plm may inspire future large language model (llm) development, although our model may not be able to exceed them in the same fine-tuning setups. our contribution is twofold: (1) we propose a new dataset for training e2e metaphor interpretation systems. to our knowledge, this is the largest manually developed, token-level metaphor paraphrase dataset. (2) we propose a plm with a novel alm paradigm for metaphor interpretation, which outperforms previous plms with similar parameter sizes. ## related work metaphor identification is a well-studied area, because of rich annotated data resources @xcite @xcite . currently, the token-level metaphor identification task is likely formulated as a sequence tagging task @xcite @xcite , using plms, e.g., bert and roberta. thus, an effective plm is important for the success of metaphor detection. metaphor interpretation is underdeveloped. @xcite interpreted metaphors by extracting properties that were shared by source and target domains. @xcite paired metaphoric sentences with handwritten paraphrases on the sentence-level. @xcite paraphrased metaphors on the token level without testing mwes. @xcite proposed an e2e model for identifying and interpreting metaphoric swes and mwes. the swe interpretation was given by the original roberta-based mwp without fine-tuning, due to the lack of labeled data. they introduced a metaphoric mwe dictionary to explain a metaphor via a clause after an original sentence. @xcite integrated a concept mapping method, based on metaphor and its paraphrase @xcite , showing the utility of such a metaphor processing paradigm in cognitive analysis @xcite @xcite . however, @xcite have some limitations including gaps between the probability distributions of the raw roberta mwp and the distributions of metaphor paraphrases in contexts. metaphor interpretation datasets were built for tasks such as definition-pairing @xcite , sentence-level paraphrases @xcite , and natural language inference (nli) testing (constitution-level) @xcite . the definitionpairing-based dataset includes metaphor definitions from dictionaries, which are independent of contexts. unlike sentence-level interpretation datasets that paraphrase the meaning of a metaphoric sentence as a whole, finer-grained interpretation datasets @xcite paraphrase metaphoric swes and mwes within contexts, and does not break up the general syntactical structures of original sentences, because the literal context words are not changed after paraphrasing. thus, finer-grained interpretation likely yields more context-consistent paraphrases and reduces the uncertainty of the learning for machines. since @xcite ; @xcite aimed to study figurative nli, each target sentence contains just an in-context swe/mwe paraphrase, which does not link well with everyday language where a sentence can have multiple metaphors. besides, these datasets include clear-cut figurative examples for nli testing, thus, they may be suboptimal for training a metaphor processing model for paraphrasing real-world texts @xcite . plms are critical for metaphor identification and interpretation. researchers developed different pretraining tasks and architectures to achieve contextualized representations for input sequences, e.g., mwp tasks (bert and roberta), language modeling tasks (gpt family), and sequence-tosequence learning tasks (t5). these plms were trained for general learning purposes, whereas the pre-training task setups were not tailored by the linguistic intuition of metaphors. for example, language modeling and sequence-to-sequence learning methods did not take bi-directional contexts into account. mwp methods used a special token "[mask]" to replace one or several original tokens to learn their representations from bi-directional contexts, whereas, mwp ignored the semantics of the original masked-out words, simply learning from their contexts. these setups are sub-optimal for the learning of metaphor processing because metaphors provide necessary semantic information in the context for their interpretation; both the forward and backward contexts help to identify metaphors and interpret their intended meanings. ## vmc-p dataset development we aim to develop a metaphor interpretation dataset that helps to yield e2e systems for real-world application scenarios. we define the interpretation task as metaphoric swe and mwe paraphrasing (tokenlevel interpretation). our dataset contains 10,716 sequences, where 50.12% of them are non-literal. we have paraphrased 11,880 metaphor units (swes and mwes). the detailed statistics and an example of our dataset can be viewed in appendix c. we source data from vmc, as it is the largest all-word annotated metaphor identification dataset. the vmc data were sourced from the everyday language with academic, fiction, news, and conversation genres. we focus on labeling open-class metaphors as they deliver richer semantic information. their paraphrases will benefit more downstream tasks. if a metaphoric mwe unit includes words with other parts-of-speech (pos), we paraphrase the mwe as a whole. metaphor identification serves interpretation in our dataset. if a metaphor cannot be paraphrased within its context without breaking the precision and conciseness of language, it is not labeled as a metaphor. we cancel 8,503 (40.2%) original metaphor labels from vmc, including 6,895 (32.6%) closed-class metaphors. metaphor interpretation is subjective and creative @xcite , which is the key annotation challenge. to improve labeling consistency, we took the following measures: first, we employed an english-speaking expert with a linguistic education background to annotate the full dataset foot_2 . thus, the only annotator can generate consistent paraphrases for metaphors 5 . second, we used wordnet @xcite hypernyms and synonyms as annotation references foot_4 . we developed a dictionary (d) that allows a user to query hypernyms and synonyms of semantic units. the annotator was asked to choose the most appropriate paraphrase lemma with the same pos as the original metaphor from d. if a suitable option was not available, the annotator then wrote a paraphrase by his/her own understanding. the references from the dictionary help the annotator generate consistent paraphrases. besides, using the dictionary as a knowledge base is computation-friendly. ## alm pre-training and fine-tuning a new plm is also proposed in this work, based on a novel alm paradigm. alm-based pre-training is inspired by two linguistic findings about metaphors: to support both metaphor identification and interpretation tasks, our pre-training employs a multi-task learning (mtl) framework to learn the anomaly detection task (a sequence tagging task corresponding to metaphor identification in finetuning) and contrastive learning task (a word classification task corresponding to metaphor interpretation in fine-tuning) together. ## experiments baselines. alm has 355m parameters. we benchmark alm to other parameter size-comparable plms, e.g., bert-large-cased (340m), robertalarge (355m), gpt2-medium (345m), and t5-base (220m), based on the mtl and stl frameworks in figures 1 and 2 . baseline hyperparameters are tuned by the results of the validation set. state-of-the-art metaphor processing models are not our direct baselines, because a) their technical novelties are irrelevant to pre-training paradigms; b) they are not foundation models for computational metaphor processing; c) our work focuses on e2e metaphor interpretation rather than metaphor identification; d) the only e2e metaphor paraphrase baseline @xcite cannot paraphrase metaphoric mwes in context and was not fine-tuned with metaphor interpretation datasets foot_7 , whereas these are the advancement of the proposed alm and vmc-p. llms are also not comparable to alm because of their huge amount of parameters, significant training, and inference costs. however, due to the absence of evaluating llms on e2e metaphor interpretation, we test gpt-3.5/4 on vmc-p in appendix e. pre-training setups. alm is pre-trained on 4 nvidia rtx a6000 gpus for 21 days. the batch size is 512. the number (n) of negative samples is 5. we use adam optimizer (kingma and ba, 2015) with a learning rate of 1e-5. the contrastive learning loss temperature (τ ) is 0.05. the pre-training loss weights (α and β) are 10 and 1, respectively. fine-tuning setups. alm is fine-tuned on an nvidia v100 gpu. the batch size is 4. the number (k) of negative samples is 3. we use adam optimizer with a learning rate of 1e-5. the contrastive learning loss temperature (τ ) is 0.05. the fine-tuning loss weights (α ′ and β ′ ) are 10 and 1, respectively. fine-tuning early stops by 5 epochs, examined on the validation set. the reported results are averaged over 5 runs. label schema. we use bio schema for metaphor identification labels. b denotes the beginning token of a bpe tokenized metaphor (swe or mwe); i denotes inside the metaphor; o denotes a literal. we use f1 as the main measure for identification and interpretation tasks (see appendix d for evaluation metrics and computational details). ## results we evaluate fine-tuned alm, termed metapro 2.0, with stl and mtl, based on vmc-p. more sequence tagging evaluation tasks can be viewed in appendix f. we show the metaphor interpretation results with gold (g) and predicted (p) identification labels. since we aim to offer useful resources for e2e metaphor interpretation, metaphor interpretation performance with predicted metaphor identification labels is the primary concern. ## conclusion we proposed a novel plm, alm, for computational metaphor processing. we also created a large metaphor interpretation dataset (vmc-p) focusing on token-level metaphor paraphrases. by finetuning alm and other parameter-size comparable plms with vmc-p, we found that fine-tuned alm (metapro 2.0) outperforms the baselines in both metaphor identification and interpretation tasks. according to the survey of @xcite , extensive research works in metaphor processing focused on metaphor identification. although it is a foundation task, the advancement of relevant tasks, e.g., linguistic and conceptual metaphor understandings have more practical values in downstream applications, e.g., enhancing machine understanding of metaphorical language @xcite , analyzing cognitive patterns of subjects through conceptual metaphor understanding @xcite @xcite , and fostering language creativity @xcite . meeting the demands of these downstream applications necessitates the development of more functional metaphor processing systems beyond mere metaphor identification. accurately paraphrasing metaphors plays a crucial role in bridging the gap between linguistic and conceptual metaphor interpretation, thereby providing an interpretable layer for text analysis @xcite @xcite . we hope that the proposed alm and vmc-p can inspire further research in this area and make a substantial contribution to the broader expansion of computational metaphor processing into downstream tasks.