Adaptation of Word-Level Benchmark Datasets for Relation-Level Metaphor Identification (2020)

Abstract: Metaphor processing and understanding has attracted the attention of many researchers recently with an increasing number of computational approaches. A common factor among these approaches is utilising existing benchmark datasets for evaluation and comparisons. The availability, quality and size of the annotated data are among the main difficulties facing the growing research area of metaphor processing. The majority of current approaches pertaining to metaphor processing concentrate on word-level processing due to data availability. On the other hand, approaches that process metaphors on the relation-level ignore the context where the metaphoric expression. This is due to the nature and format of the available data. Word-level annotation is poorly grounded theoretically and is harder to use in downstream tasks such as metaphor interpretation. The conversion from word-level to relation-level annotation is non-trivial. In this work, we attempt to fill this research gap by adapting three benchmark datasets, namely the VU Amsterdam metaphor corpus, the TroFi dataset and the TSV dataset, to suit relation-level metaphor identification. We publish the adapted datasets to facilitate future research in relation-level metaphor processing.

## introduction metaphor is a ubiquitous figurative device that represents the interaction between cognition and language @xcite . a metaphor contains an implied analogy where a concept (represented by a word sense) is borrowed to represent another concept by exploiting common or single properties of both concepts. generally, a metaphor has two main components, the tenor and the vehicle; the relation between them is called the ground. the tenor represents the topic of the metaphor while the vehicle is the term used metaphorically and the ground gives the metaphor its meaning @xcite . perceiving these components is essential to fully comprehend the metaphor. in this work, we adopt the conceptual metaphor theory (cmt) by @xcite to view metaphor where there is an underlying mapping between a source domain (the vehicle) and a target domain (the tenor). for example, a concept such as "fragile object" (source domain/vehicle) can be borrowed to express another such as "emotions" (target domain/tenor). this conceptual metaphor "emotions are fragile objects" can be expressed in our everyday language in terms of linguistic metaphors such as "shattered my emotions","break his soul","crushed her happiness", "fragile emotions" and "brittle feelings". due to their nebulous nature, metaphors are quite challenging to comprehend and process by humans, let alone computational models. this intrigued many researchers to develop various automatic techniques to process metaphor in text. metaphor processing has many potential applications, either as part of natural language processing (nlp) tasks such as machine translation @xcite ), text simplification @xcite and sentiment analysis @xcite or in more general discourse analysis use cases such as in analysing political discourse @xcite , financial reporting @xcite and health communication @xcite . the computational processing of metaphors can be divided into two tasks, namely metaphor identification and its interpretation. while the former is concerned with recognising the metaphoric word or expression in a given sentence, the latter focuses on discerning the meaning of the metaphor. metaphor identification is studied more extensively than metaphor interpretation, in part due to the availability of datasets. identifying metaphors in text can be done on either the sentence, grammatical relation or word levels. sentence-level approaches classify the whole sentence that contains the metaphoric word/expression without explicit annotation of the source and target domain words. relation-level metaphor identification focuses on certain grammatical relations by looking at pairs of words where both the source and target domain words are classified as a metaphoric expression. it is also referred to as phrase-level metaphor identification due to the way a sentence is divided into sub-phrases with various syntactic structures (we use these two terms indistinguishably in the context of this paper). the most commonly studied grammatical relations are verb-noun and adjectivenoun relations where the metaphoricity of the verb or the adjective (source domain/vehicle) is discerned given its association with the noun (target domain/tenor). finally, word-level metaphor identification approaches treat the task as either sequence labelling or single-word classification. in both methods, only the source domain words (vehicle) are labelled either as metaphoric or literal given the context. many approaches are designed to identify metaphors of different syntactic types on the word-level but the most frequently studied ones are verbs. in this paper, we are interested in relation-level metaphor identification focusing on the data availability for this level of processing. the next section explains, in detail, the difference between wordlevel and relation-level metaphor analysis highlighting the research gap that we aim to tackle. ## related work this work is inspired by @xcite and @xcite who attempted to adapt existing word-level metaphor identification datasets to suit their relation-level (phrase-level) identification approaches. @xcite was the first to create an annotated dataset for relation-level metaphor identification. the robust accurate statistical parsing (rasp) parser @xcite was utilised to extract verb-subject and verb-direct object grammar relations from the british national corpus (bnc) @xcite . the dataset comprises around 62 verb-noun pairs of metaphoric expressions, where the verb is used metaphorically given the complement noun (tenor). the trofi dataset, which was designed to classify particular literal and metaphoric verbs on the word-level, was adapted by @xcite in order to extract metaphoric expressions on the relation-level. the authors parsed the original dataset using the turbo dependency parser @xcite to extract subject-verb-object (svo) grammar relations. the final dataset consists of 953 metaphorical and 656 literal instances. in the same work, tsvetkov et al. also prepared a relationlevel metaphor identification dataset, referred to as the tsv dataset, focusing on adjective-noun grammar relations. we will further describe this dataset in section 4. more recently, @xcite adapted the benchmark moh dataset, which was initially created to extract metaphoric verbs on the wordlevel, to suit relation-level metaphor identification of verb-noun relations. verb-direct object and verbs-subject dependencies were extracted and filtered yielding a dataset of 647 verb-noun pairs, out of which 316 instances are metaphorical and 331 instances are literal. to the best of our knowledge, there is no attempt to adapt the benchmark vu amsterdam metaphor corpus, referred to as vuamc, to suit relation-level metaphor identification. this has discouraged other researchers focusing on relationlevel approaches to employ this dataset such as the work done by @xcite al. (2017), bulat et al. (2017), shutova et al. (2016) and tsvetkov et al. (2014) who did not evaluate or compare their approaches using this dataset. in this paper, we introduce the first adapted version of the vuamc. furthermore, we adapt the trofi and the tsv datasets to better suit relation-level metaphor processing. ## datasets as mentioned in section 2, the widely used benchmark datasets for word-level metaphor identification are trofi, vuamc and moh datasets, while tsv, shutova's adaptation of moh and zayed's tweets datasets are commonly used for relationlevel metaphor identification. the vu amsterdam metaphor corpus (vuamc) foot_0 , introduced by @xcite , has become one of the most well-known metaphor corpus existing nowadays. it is the largest corpus annotated for metaphors and has been used extensively to train, evaluate and compare models that identify metaphors on the word-level. the corpus consists of 117 randomly selected texts from the bnc baby version which comprises various text genres, namely academic, conversation, fiction and news. the corpus is annotated for metaphors on the word-level, regardless of the word's syntactic type, through a collaborative annotation scheme. the employed annotation scheme is referred to as the metaphor identification procedure (mipvu) by which a strong inter-annotator agreement of 0.84 is obtained, in terms of fleiss' kappa @xcite , among four annotators. the dataset is published in an xml format; figure 2 shows an example of the corpus where the metaphoric words are tagged as @xmath0. the naacl 2018 metaphor shared task @xcite employed the vuamc in order to develop, train and test systems to identify metaphors on the word-level. the shared task consisted of two tracks, which are 1) all part-of-speech (pos) to identify nouns, verbs, adverbs and adjectives that are labelled as metaphorical; 2) verbs track which is concerned only with identifying metaphorical verbs. all forms of the verbs: "be, do, and have" are excluded for both tracks. the corpus is then divided into training and test sets according to the focus of each track. a script is provided to parse the original vuamc.xml file foot_1 which contains the corpus, since the corpus is not directly downloadable due to licensing restrictions. in this paper, we utilise the dataset from the verbs track from this shared task. the main limitation of the vuamc, and any dataset that stems from it, is that it only suits the identification of metaphors on the word-level. thus, it is not possible to apply the vuamc in its current state to relation-level metaphor identification and there are no larger dataset designated to support relation-level metaphor identification since the size of shutova's adaptation of moh and zayed's tweets datasets is relatively small for training state-of-the-art neural models. the trofi dataset is one of the earliest metaphor identification datasets introduced by @xcite . the dataset focuses on the metaphoric usage of 50 selected verbs and comprises 3,727 english sentences extracted from the 1987-1989 wall street journal (wsj) corpus. the metaphoricity of the selected verbs on the word-level is identified by manual annotation. the inter-annotator agreement was calculated on a random sample of 200 annotated sentences scoring 0.77 in terms of cohen's kappa @xcite among two annotators. the dataset had been used to evaluate the performance of many word-level metaphor identification systems. in order to use this dataset for relation-level metaphor identification of verb-noun relations, further annotation is required to highlight the complementing noun (tenor) of each metaphoric verb (vehicle). the tsv dataset @xcite was created to support relation-level metaphor identification approaches that focus on adjective-noun grammatical relations. the dataset comprises ∼2,000 adjective-noun pairs which were selected manually from collections of metaphors on the web. it is divided into 1,768 pairs as a train set and 200 pairs as a test set. as mentioned earlier, only the test set contains the full sentences which was obtained from the english ten-ten web corpus @xcite by utilising sketchengine foot_2 @xcite . the annotation scheme depended on the intuition of the human annotators to define the metaphoric expressions. an inter-annotator agreement of 0.76, in terms of fleiss' kappa, was obtained among five annotators on the test set. the main limitation of this dataset is the absence of the full sentences in the training set which forces the models employing it to either ignore the context that surrounds the adjective-noun pairs or to use the small test set in a cross-validation experimental setting which makes the model prone to overfitting. ## dataset adaptation methodology in this section, we discuss the methodology of adapting the vuamc, trofi and tsv datasets to better suit relation-level (phrase-level) metaphor processing. ## quality assessment and enhancement in order to assess the quality of the adapted datasets, we suggested a preliminary quality assessment scheme and tested it through an initial experiment on a randomly sampled subset from each dataset. we then employed this scheme to ensure the quality of the whole datasets. ## conclusion and future work in this paper, we took a step towards filling the gap of the availability of large benchmark datasets for relation-level metaphor processing in english text by utilising existing word-level datasets. we employed a semi-automatic approach to adapt the vuamc to better suit identifying metaphors on the relation-level without the need for extensive manual annotation. we also adapted the trofi dataset, one of the earliest word-level datasets for metaphor identification of verbs, to support verb-noun metaphor identification. furthermore, we extended the tsv dataset which was originally annotated on the relation-level focusing on dataset total % accepted by annotator tsv the tweet is in understandable english? 70% the relation is syntactically valid? 82.75% did the context (tweet) kept the metaphoric sense of the expression? 99.36% trofi the relation is syntactically valid? 98.52% the verb is metaphoric or literal due to the associated object? 100% vuamc the relation is syntactically valid? 98.42% the verb is metaphoric or literal due to the associated object? 98.5% annotation disagreement or inconsistency 5.45% in future work, we will expand the adapted vuamc to include verb-subject (i.e. nsubj) and adjective-noun (i.e. amod) relations. moreover, we plan to consolidate these adapted datasets in one repository categorised by data source and text genre. we also plan to invest extra annotation effort to ensure the consistency of the annotated instances across the different datasets using weakly supervised approaches.