Testing the role of metadata in metaphor identification (2020)

Abstract: This paper describes the adaptation and application of a neural network system for the automatic detection of metaphors. The LSTM BiRNN system participated in the shared task of metaphor identification that was part of the Second Workshop of Figurative Language Processing (FigLang2020) held at the Annual Conference of the Association for Computational Linguistics (ACL2020). The particular focus of our approach is on the potential influence that the metadata given in the ETS Corpus of Non-Native Written English might have on the automatic detection of metaphors in this dataset. The article first discusses the annotated ETS learner data, highlighting some of its peculiarities and inherent biases of metaphor use. A series of evaluations follow in order to test whether specific metadata influence the system performance in the task of automatic metaphor identification. The system is available under the APLv2 open-source license.

## introduction research on metaphors, particularly in the framework of conceptual metaphor theory, continues to grow in all genres of language use and across diverse disciplines of linguistics (cf., among others, @xcite @xcite @xcite degani for recent and forthcoming overviews and extensions, and @xcite for a book-length discussion of computational linguistic perspectives). while the importance of metaphor in thought and everyday language use has long been acknowledged @xcite , the practice of metaphor research still faces two methodological and analytical challenges: first of all, the identification of metaphors and, secondly, their actual description through source and target domains. in computational linguistics, a great amount of recent work has been concerned with addressing the challenge of identifying metaphors in texts. this is evident in the series of four workshops on metaphor in nlp from 2013 to 2016 and in the two workshops on figurative language processing in 2018 @xcite and 2020 @xcite , each of which involved a shared task (st) in automatic metaphor detection. identification systems that achieved the best results in the first shared task relied on neural networks incorporating long-term short-term memory (lstm) architectures (see @xcite for a discussion). further advances in the field using deep learning approaches have been reported in @xcite , @xcite , and @xcite . this paper extends a system proposed in @xcite , which combines word embeddings (wes) of corpora like the bnc (british national corpus consortium, 2007) and the toefl11 language learner corpus (see @xcite . with the modified system, we participated in the second shared task on metaphor detection. the difference to the 2018 edition of the st is a new set of data. as in the first task, one part of the dataset is based on the vu amsterdam (vua) metaphor corpus manually annotated according to the mipvu procedure @xcite . additionally, the second task includes a sample of 240 argumentative learner texts. these texts are taken from the ets corpus of non-native written english (synonymous to the toefl11 corpus) and have been manually annotated @xcite . since the learner essays are a very specific kind of data, the aim of this study is to build upon observations from @xcite , who found that a combination of word embeddings from the bnc and the toefl11 learner corpus yielded the best results of metaphor identification in a bi-directional recursive neural network (birnn) with lstm. these results triggered the hypothesis that learner language can lead to an information gain for neural network based metaphor identification. to explore this hypothesis further, the current study puts an explicit focus on the metadata provided for the ets corpus of non-native written english and specifically tests the potential influence of proficiency ratings, essay prompt, and the first language (l1) of the author. in addition, we also test whether a combined training on the diverse datasets and the sequence of this training will have an impact on our system of neural network based metaphor identification. to address these aims, our paper is structured as follows: section 2 provides observations on the annotated learner corpus dataset. section 3 describes the system of metaphor identification. this is followed in section 4 by the results of the experiments, which are briefly discussed in light of the observations on the annotated learner corpus data. ## observations on the data the vua metaphor corpus and its application in the first shared task has been concisely described in @xcite . the authors have reported the relatively high inter-annotator agreement (κ > 0.8), which is in part due to the mipvu protocol @xcite and the close training of annotators in the amsterdam metaphor group. interestingly, the results of the first task across all submitted systems showed a clear genre bias with academic texts consistently displaying the highest correct identification rates and conversation data (i.e. spoken texts) the lowest @xcite . this might be related to the fact that academic discourse is more schematic and formulaic (e.g. in the use of sentential adverbials and verbal constructions) and might rely to a greater extent on recurrent metaphorical expressions than spoken conversations, which are less schematic and can thus display a higher degree of syntactic and lexical variation. in other words, similarities in the data between training and test sets might be higher in the academic than in the conversation genre, leading to different genre-specific training effects in neural networks. apart from the vua metaphor corpus, the second shared task introduces a novel dataset culled from the ets corpus of non-native written english. in their description, beigman klebanov et al. the relatively small size of the ets sample inspired us to look into the structure of the data more closely to check whether any potential biases exist that might play a role for the automatic detection of metaphors. beigman @xcite report a significant positive correlation of the number of metaphors and the proficiency ratings of the texts as medium or high in the data. this relationship is confirmed by an independent samples t-test in the training partition of the data (180 texts). the group of highly proficient learners (@xmath0.98, @xmath1.23) compared to the group of medium proficient learners (@xmath2, @xmath3.55, @xmath4.96) at a significantly higher rate (@xmath5.07, @xmath6.000071). the l1 background of the learners (i.e. arabic, italian, japanese) did not influence the mean number of metaphors in the texts as confirmed by a oneway anova (@xmath7.619, @xmath8.201; l1 arabic: @xmath9, @xmath10.48, @xmath11.37; l1 italian: @xmath12, @xmath13.71, @xmath14.79; l1 japanese: @xmath15, @xmath16.44, @xmath17.38). since the corpus of non-native written english consists of argumentative learner essays that were written in response to one of eight different prompts, another factor to consider in the annotated ets sample is the role the prompt might have on metaphor use. port, per prompt, the total number of words, the number of metaphor types, the overall number of metaphor tokens (i.e. all words annotated for metaphor), the type token ratio of metaphors, the relative amount of metaphor tokens among all words, and the mean values of metaphors. the two rightmost columns illustrate an uneven occurrence of metaphors across the diverse prompts. three groups emerge from the data according to their similarly high (or low) values: p1, p2, p3 as the highest scoring group, p4, p5, p6 as the lowest scoring group, and p7, p8 whose values are in-between the other two groups. a one-way anova for independent samples (@xmath18.0919, @xmath19.00001) confirms a significant difference between the groups. t-tests comparing the minimal and maximal numerical distances between the high, the medium, and the low clusters show that the high cluster is significantly different from the low cluster (p1: @xmath20, @xmath21.70, @xmath22.138 compared to p5: @xmath23, @xmath24.96, @xmath25.117) at t(44) = 3.54, @xmath26.000948. the differences between the low and the medium groups as well as the high and the medium group do not reach a significance threshold of @xmath27.01. when looking for an explanation of these biased metaphor occurrences, some interesting patterns emerge among the high frequency group (p1, p2, and p3). in all these instances, the prompts trigger certain metaphor-related words (mrw) that occur at a high rate. in in all the other prompts, trigger effects do not occur or are not as quantitatively relevant as in p1 to p3. p4, for example, does not show any spikes in metaphor frequencies with the most frequent mrw (image) merely occurring 5 times. the same is true in p5 with the terms ruining, reach, comfortable, and advancement being mentioned 4 times each as the most frequent mrw. a weak effect can be observed in p6 where the prompt "the best way to travel is in a group led by a tour guide" triggers the metaphorical collocation to "take a trip" that recurs 14 times across the learner texts. in p7 and p8, the most frequent mrw are not stimulated by the prompt, and there are similarly low token frequencies leading to a flat frequency distribution among the mrw. incidentally, p8 ("successful people try new things and take risks rather than only p1: it is better to have broad knowledge of many academic subjects than to specialize in one specific subject. p2: young people enjoy life more than older people do. doing what they already know how to do well") contains the metaphorical expression "take risks" that recurs in the learner essays 43 times. however, it has not been manually annotated as an mrw in the ets corpus. if we take a look at how the mrw are distributed across the different parts of speech, it is interesting to note that verbs are by far more often marked as metaphors than nouns and adjectives/adverbs. among the 30 most frequent mrw per prompt in the training set, 416 metaphorical verbs precede over 198 adjectives/adverbs and 141 nouns. finally, the learner data poses another peculiarity that is worth considering in the automatic metaphor identification. there is a total of 99 misspelled mrw across all prompts in the data (4.6% of all mrw) as in messege, actractivity, strenght, knowled, isolte, dangerousness, and broadn to randomly pick out a few. finding a way to factor in the out-of-vocabulary words will increase the performance of automatic metaphor detection. ## design and implementation our approach extends the system from @xcite , which combines a small set of established machine learning (ml) methods and tools. in particular, the system uses fasttext foot_0 word embeddings from different corpora in a bi-directional recursive neural network architecture with long-term short-term memory (lstm birnn) and implements a flat sequenceto-sequence neural network with one hidden layer using tensorflow+keras @xcite in python. with the goals we introduced in section 1, it seemed sufficient to use a successful system from the last st instead of improving the overall system by integrating the latest, highly successful 2 developments from the field of nlp (see, e.g., wang et al., 2020 for an overview). in their experimental design stemle and onysko (2018) use word embeddings from corpora such as wikipedia foot_2 and bnc foot_3 as well as from texts of language learners (toefl11). this is to follow the intuition that the use of metaphors can vary depending on the competence of the learners and that these differences can be helpful in training a metaphor detection system. for the current st, the system was slightly extended. we for the last point, we adapted the way the input for the neural network is represented: the number of input layers corresponds to the number of features, i.e. for multiple features, e.g. multiple we models or additional pos tags, sequences are concatenated on the word level such that the number of features for an individual word grows. for the metadata, we added an input layer with the number of dimensions varying with the number of encoded metadata. we maintain the implementation in a source code repository foot_6 . the system is available under the aplv2 open-source license. ## conclusion this paper has focused on the structure of the learner data used in the second shared task of metaphor identification. we aimed at exploring possible factors that influence this kind of data and tested whether these play a role for the automated identification using word embeddings in an established lstm birnn system from the first st in 2018. a descriptive investigation of the manually annotated sample of the ets corpus of non-native written english (toefl11 corpus) shows that the factors of proficiency and especially the essay prompt exhibit significant correlations to the amount and type of metaphors found in the annotated training set. the data also show a numerical bias towards the annotation of verbs as metaphors compared to other content words. a sequential training of the bidirectional neural network using both the vua and the toefl partitions of the shared task points to the different structure of the datasets, in particular towards an emerging bias of overidentifying verbal metaphors in the neural network based classification. the hypothesized influence of the metadata in the toefl set, in particular the observed dependencies on proficiency and the essay prompt, was not confirmed by the results of the automated identification. while the factors of l1, proficiency, prompt and essay length did not influence the baseline results, the essay id (i.e. the individual learner) reduced the performance of the system as did a combination of all metadata. for the future, more tests with different ways of modelling the metadata in the neural network architecture and on the test set of the task will provide further insights.