Neural Metaphor Detection with a Residual bi LSTM - CRF Model (2020)

Abstract: In this paper we present a novel resource-inexpensive architecture for metaphor detection based on a residual bidirectional long short-term memory and conditional random fields. Current approaches on this task rely on deep neural networks to identify metaphorical words, using additional linguistic features or word embeddings. We evaluate our proposed approach using different model configurations that combine embeddings, part of speech tags, and semantically disambiguated synonym sets. This evaluation process was performed using the training and testing partitions of the VU Amsterdam Metaphor Corpus. We use this method of evaluation as reference to compare the results with other current neural approaches for this task that implement similar neural architectures and features, and that were evaluated using this corpus. Results show that our system achieves competitive results with a simpler architecture compared to previous approaches.

## introduction this paper presents a new model for automatic metaphor detection which has participated at the figlang 2020 metaphor detection shared task @xcite . our approach, which is based on neural networks, has been developed in the framework of the research project moment @xcite , a project devoted to the analysis of metaphors in mental health discourses. as it is well known in cognitive linguistics, a conceptual metaphor (cm) is a cognitive process which allows to understand and communicate an abstract or diffuse concept in terms of a more concrete one (cf. e.g. @xcite ). this process is expressed linguistically by using metaphorically used words (muw). the study of metaphor is a prolific area of research in cognitive linguistics, being the metaphor identification procedure (mip) @xcite and its derivative mipvu @xcite the most standard methods for manual muw detection. mipvu is the method that was used to annotate the vu amsterdam metaphor corpus (vua corpus), used in figlang 2020. moreover, in the area of corpus linguistics, some methods have been developed for a richer annotation of metaphor in corpora @xcite @xcite . cm is pervasive in natural language text and therefore it is crucial in automatic text understanding @xcite . for this reason automated metaphor processing has become an increasingly important concern in natural language processing, as shown by the holding of the metaphor in nlp workshop series @xcite @xcite @xcite and a growing body of research -see @xcite and @xcite for quite recent reviews. automatic metaphor processing involves two main tasks: identifying muw (metaphor detection or recognition) and attempting to provide a semantic interpretation for the utterance containing them (metaphor interpretation). this work deals with metaphor detection. this problem has been mainly approached in the last decade by supervised and semi-supervised machine learning techniques but recently this paradigm has largely shifted to the use of deep learning algorithms, such as neural networks. @xcite report that all but one of participating teams on the 2018 vua metaphor detection shared task used this kind of architectures. our system follows this trend by trying to improve on previous neural network methods. below we describe the main related works (section 2). next we present our methodology and model (section 3), experiments (section 4) and results (section 5). we finish with the discussion and our overall conclusions (sections 6 and 7). ## background research on metaphor recognition and interpretation is changing from the use of features (linguistic and concreteness features), classical methods (as generalization, classification and word associations) and the use of theoretical principles (construction grammar, frame semantics and conceptual metaphor theory) to neural networks and other deep learning techniques. concreteness features are used by @xcite along with re-weighting of the training examples to train a supervised machine learning system. the trained system is able to classify all content words of a text in two groups: metaphorical and non-metaphorical. @xcite study the metaphoricity of verbs using semantic generalization and classification using word forms, lemmas and several other linguistic features. they demonstrated the effectiveness of the generalization from orthographic unigrams to lemmas and the combination of lemmas and semantic classes based on wordnet. they also used automatically generated clusters to combine with unigram lemmas getting a competitive performance. the meta4meaning @xcite metaphor interpretation method uses word associations extracted from a corpus to retrieve approximate properties of concepts and provide interpretations for nominal metaphors of the form noun 1 is [a] noun 2 (where noun 1 is the tenor and noun 2 the vehicle). metaphor interpretation is obtained as a combination of the saliences of the properties to the tenor and the vehicle. combinations can be aggregations (the product or sum of saliences), salience difference or a combination of the results of the two. as an output, meta4meaning provides a list of interpretations with weights. the automatic metaphor detection system metanet @xcite has been designed applying theoretical principles from construction grammar, frame semantics, and conceptual metaphor theory. the system relies on a conceptual network of frames and metaphors. @xcite developed an algorithm using deep learning techniques that uses a representation of metaphorical constructions in an argument-structure level. the algorithm allows for the identification of source-level mappings of metaphors. the author concludes that the use of deep learning algorithms with the addition of construction grammatical relations in the feature set improves the accuracy of the prediction of metaphorical source domains. @xcite propose to use a convolutional neural network -long-short term memory (cnn-lstm) with a conditional random field (crf) or softmax layer for metaphor detection in texts. they combine cnn and lstm to capture both local and long-distance contextual information to represent the input sentences. meanwhile, @xcite argue that using broader discourse features can have a substantial positive impact for the task of metaphorical identification. they obtain significant results using document embeddings methods to represent an utterance and its surrounding discourse. with this material a gradient boosting classifier is trained. other works for specific tasks within the scope of metaphor recognition, such as detecting the metaphoricity of adjective-noun (an) pairs in english as isolated units, include the works by @xcite , @xcite . the main goal of this task is to classify an collocations using external and internal linguistic features, or techniques such as transfer learning along with word embeddings. we propose a model that uses residual bidirectional long short-term memory (bilstm) with a crf, using elmo embeddings along with additional linguistic features, such as part of speech tags (pos) and semantically disambiguated word-net foot_0 synonym sets (synsets) @xcite . our model could be grouped in the same category as the aforementioned approaches: deep neural networks models for metaphor detection. ## model description most of the approaches mentioned in section 2 used the vua corpus @xcite in order to carry out model training and testing. they divided the training and test sets according to the vua metaphor detection shared task specifications. to train and test our model we used the vua corpus partitions, using elmo embeddings to represent words and lemmas, and pos and synsets as additional linguistic features. elmo (embeddings from language models) embeddings @xcite are derived from a bidirectional language model (bilm) and they are contextualized, deep and character based. elmo embeddings have been successfully used in several nlp tasks. to process the vua corpus we used the natural language toolkit (nltk) @xcite for python, with this tool we performed tokenization, lemmatization, and pos tagging. then we used freeling @xcite to obtain the respective synset of each token. although nltk provides a method for obtaining synsetsusing pos tags or lesk's algorithm-, freeling implements ukb @xcite , a graphbased word sense disambiguation (wsd) algorithm that is used to obtain semantically disambiguated synsets. these features along the elmo embeddings were used -in different configurations-as input for our model. we set a sequence padding value equal to 116, which is the maximum sentence length observed in the corpus. this process normalizes the input in order to train in batches, but might contribute to sparsity on training data. we used one-hot encoded representation for pos, and computed local 100-dimension embeddings for synsets. in the case of pos, we have a small set of tags (43), and therefore resulting in a low dimensionality of the one-hot embeddings. for synsets, the computation of local embeddings provides the semantically disambiguated relations that exist between the units that compose the training data. these embeddings, in addition with their emlo counterparts, shall provide enough contextual and semantic data to understand metaphorical instances of words. the main architecture of our model (shown in figure 1 ) is composed by a residual bilstm @xcite for sequence labeling. one of the particularities of this architecture lies in the implementation of an additive operation that takes the outputs from each bilstm layer and combines them to calculate the residual connection between them, in order to obtain previously seen information from both instances. after computing the residual connection from both bilstm layers, our model includes a dropout layer, followed by a time distributed layer in which a dense activation with 2 hidden units to each timestep is applied. we used relu @xcite as activation function in combination with a he-normal @xcite kernel initialization function for the time distributed layer, which results in a zero-mean gaussian distribution with a standard deviation equal to 2 nl . finally, after the time distributed layer we used a conditional random field (crf) implemented for sequence labeling @xcite . given that the vua corpus is composed by more negative -or literal-labels than positive -or metaphoric-labels, and that the sequence padding process added non-informative features to the input array, we opted to treat the training partition as an imbalanced dataset. we selected the nadam optimizer @xcite , which is based on adam (kingma and ba, 2014) and tends to perform better with sparse data. this last optimization algorithm has two main components: a momentum and an adaptive learning rate component. nadam modifies the momentum component of adam using nesterov's accelerated gradient (nag). the nadam update rule can be written as follows: to carry out the evaluation of our model we used the train and test splits provided in vua shared task partitions @xcite . in order to obtain a validation split we divided the training partition using the following percentages: 80% for training 20% for validation. with these partitions, we trained a total of 6 different model configurations: words and pos (w+pos); lemmas and pos (l+pos); words, pos and synsets (w+pos+ss); lemmas, pos and synsets (l+pos+ss); words, lemmas and pos (wl+pos); and words, lemmas, pos and synsets (wl+pos+ss). in all cases we used the same training parameters, all model configurations were trained in batches for 5 epochs, using a learning @xmath0.0025. then, the resulting models were evaluated -using the precision, recall and f 1 score metrics-on both the all pos metaphor detection task and the metaphoric verbs detection task. ## discussion our proposed architecture has similarities to other current approaches such as @xcite who propose a lstm with softmax model, and @xcite who implement an xgboost classifier using elmo embeddings. in comparison to these approaches, our model shows an improvement in precision on the verb labeling task with a value equal to 0.6695, while @xcite reported a precision score of 0.600 foot_2 , and @xcite a precision equal to 0.589. nevertheless @xcite reported the highest f 1 score (0.671), and @xcite the highest recall (0.771). ## conclusions and further work in this paper we have described the system we have presented at the figlang 2020 metaphor detection shared task. our approach is based on neural networks using a residual bilstm with a crf and using elmo embeddings along with the inclusion of several combinations of words, lemmas and linguistic features as pos and wordnet synsets. the system achieves competitive results with a simpler architecture compared to systems found in the literature. such systems implement similar elements such as the use of bidirectional lstm, crf and elmo embeddings in different configurations, and with different combination of linguistic features. as future work, we plan to further analyse which pos benefits most from the inclusion of synset information. other aspect we want to explore is how to deal with imbalanced data, i.e. how we can leverage a dataset with only two classes (metaphoric/literal) where most of the samples are literal. other interesting questions that deserve more research is the effects on optimal dimensionality of the addition of linguistic information. other features that could be implemented are concreteness value of certain words, or as an strategy to balance classes according to the influence that this feature has on literal and metaphoric classes. other future lines of work might include the implementation of this type of model for the detection of metaphors and source domain identification in spanish. current developments on metaphor detection are being carried out mainly in english, while this is a great resource it could be interesting to create resources in other languages to broaden the scope of metaphor detection and interpretation. a possible pipeline could be configured with two separated model, one that performs the detection of metaphorical words, followed by another classifier that predicts the domain of those metaphors.