Being neighbourly: Neural metaphor identification in discourse (2020)

Abstract: Existing approaches to metaphor processing typically rely on local features, such as immediate lexico-syntactic contexts or information within a given sentence. However, a large body of corpus-linguistic research suggests that situational information and broader discourse properties influence metaphor production and comprehension. In this paper, we present the first neural metaphor processing architecture that models a broader discourse through the use of attention mechanisms. Our models advance the state of the art on the all POS track of the 2018 VU Amsterdam metaphor identification task. The inclusion of discourse-level information yields further significant improvements.

## introduction metaphor widely manifests itself in natural language. it is used to make an implicit comparison between two distinct domains that have certain common aspects @xcite . for instance, in the sentence "the price of the commodity is rising", the target domain of quantity (price) can be understood through the source domain of directionality (rising). the majority of computational approaches to metaphor focus on the task of its identification in text. early approaches utilised hand-crafted features based on word classes @xcite , concreteness and imageability ratings @xcite or selectional preferences @xcite . succeeding research has moved on to corpus-based techniques, such as the use of distributional and vector space models @xcite @xcite , and more recently, deep learning methods @xcite . current metaphor identification approaches cast the problem in the sequence labelling paradigm and apply convolutional @xcite , recurrent @xcite @xcite and transformer-based neural models @xcite . however, these approaches model only local linguistic context, i.e. information about the sentence in which the metaphor resides. yet, a large body of corpus-linguistic research suggests that metaphor production and comprehension is influenced by situational information and wider discourse properties @xcite @xcite . previously presented computational models of metaphor incorporating discourse use handcrafted features @xcite or neural sentence embeddings @xcite within a simple classification paradigm. since these methods employ shallow classification models, their task performance is subpar compared to deep neural architectures. nonetheless, these studies established that discourse-level information is beneficial for metaphor detection. improving upon prior methods, we present a novel neural metaphor identification architecture that incorporates broader discourse. to model discourse, we investigate two types of attention mechanisms: a shallow general attention mechanism and a hierarchical one @xcite . the former builds a sentence representation by applying wordlevel attention. the latter combines attention at both word and sentence level. we apply our models to the 2018 vu amsterdam (vua) metaphor identification shared task @xcite , specifically to the all pos subtask. this task involves metaphor detection for all open-class words -i.e. verbs, adjectives, nouns and adverbs. our results confirm that modelling discourse is beneficial for metaphor detection and our models improve upon the previous state-of-the-art task performance @xcite ) by 5.2 and 6.4 f1-points, for our bestperforming baseline and discourse models, respec-tively. to the best of our knowledge, this is the first end-to-end neural approach investigating the effect of broader discourse on metaphor identification. ## data the task revolves around performing binary classification -identifying whether a word is metaphorical or literal -on the all pos subtask of the 2018 vua metaphor identification shared task. this task uses a dataset consisting of 117 text excerpts from the british national corpus @xcite , labelled in the vua metaphor corpus @xcite . each excerpt has been retrieved from one of the following four genres: academic, news, conver- label attention classification encoder embedding l ⊕ preceding sentence focus sentence succeeding sentence . . . . . . . . . . . . . . . . . . . . . . . . . . . ⊕ l m (a) general attention preceding sentence focus sentence succeeding sentence . . . . . . . . . ⊕ . . . . . . . . . ⊕ . . . . . . . . . sation, and fiction. metaphorical expressions are annotated at word level. following the shared task setup, we evaluate our models on open-class words only -i.e. verbs, adjectives, nouns and adverbs. the dataset contains 72,611 and 22,196 labelled words in the training and test set, respectively. 15% and 18% of these words are metaphorical for the two sets, respectively. we randomly sample 10% of the training data texts for validation purposes. ## methods we construct a neural architecture that is optimised to predict binary metaphoricity at word level, by embedding the input words, applying encoder layers followed by a classification layer and softmax activation to yield per-word predictions. we present two variants of the model. the first method is feature-based, similar to the model of @xcite . we embed input words through the concatenation of their non-contextualised (glove) and contextualised (elmo) embeddings. the concatenation of glove and elmo feeds into a one-layer bi-lstm encoder. during optimisation, we learn the parameters of the bi-lstm and classification layer. as our second method, we use a fine-tuning architecture, similar to @xcite , in which the embeddings and recurrent encoder are replaced with the pretrained bert base model @xcite . bert uses embeddings for subword units that are encoded with twelve transformer en-coder layers. during optimisation we fine-tune bert and learn the parameters of the classification layer. a word is considered metaphorical if any of its subword units is labelled as metaphorical. this choice is based on the assumption that it is more likely that a common prefix or suffix is not considered metaphorical while a word's main piece is than the other way around. to include discourse information in both architectures, the output of the last encoder layer is concatenated to a discourse representation and fed to the classification layer, such that the classification layer contains dedicated parameters for both the discourse and (sub)word representations. we further detail the two mechanisms employed to compute the discourse representations below. ## analysis and discussion in order to investigate the types of information that discourse provides, we have conducted a qualitative analysis of the sentences where our discourseaware models improved the labelling of a word over the discourse-agnostic baselines. we found that discourse helps primarily in two ways: (1) it provides information about the topic of the text, which is often needed for disambiguation, particularly for shorter sentences; and (2) it allows the models to implicitly perform co-reference resolution through the use of word-level attention. we observe that while the attention distributions of the hierarchical mechanisms are rather diffused, the general attention mechanism uses very sparse weights. we hypothesise that the former type is more suited for modelling coarse-grained topical information. the latter is able to highlight specific words and phrases in neighbouring sentences that may be needed for disambiguation and coreference resolution. the attention distributions displayed in to further investigate the internal functioning of the attention modules and the influence of discourse, we use majority voting across the randomly initialised runs to obtain samples that are consistently improved by all discourse setups compared to the baseline per model type. for the recurrent models, this subset contains 109 samples: 53 metaphorical tokens and 56 literal ones. for bert, these statistics are 57 and 19, respectively. to assert that within these subsets, the surrounding sentences affect the discourse representation, we visualise the average weight per word position, as measured from the middle of the focus sentence. figure 2 demonstrates that for sentences in this subset, the distribution is more diffused compared to all test sentences. thus, on average, the broader discourse has a more substantial effect on the model's classification for these samples. this finding supports the hypothesis that the numerical performance gain observed is caused by the inclusion of discourse representations. the examples listed in hierarchical attention general attention sentence word .297 i 've got l some cooking l apples l out there oh is n't he , i could hit l him ! apples l .419 why does n't he make l make l your own bloody m mincemeat l then ! bloody m, mincemeat l .284 yeah that 's cos l i make l the the pastry l and you can -.276 as you always l still l continue l to tell l them yes l you do . -.321 if i want l it spread m around . spread m, around .403 that gives m you your bit m of character m . gives m, bit m, character m .344 the accelerator l is the one on the right l sir l ! -.268 he is an old l gentleman l , dear l but er l he is n't in a hurry l what so ever l . hurry l .388 look l he 's slowing l down there ! look l, slowing l, down in the final example, it becomes apparent from the metaphor's discourse context that "being in a hurry" in fact concerns the actual speed of the accelerator, as opposed to the metaphorical use of hurrying: the obstacle that keeps us all from living life fully. the results in ## conclusion in this work, we presented deep neural architectures that make use of attention mechanisms to investigate the impact of discourse in the task of word-level metaphor detection. our models establish new state-of-the-art results in the all pos track of the 2018 vua metaphor identification shared task @xcite . two attention mechanisms were experimented with for modelling discourse, a general and hierarchical one. both modules yield significant performance increases, but our qualitative analysis indicates that they serve a different purpose. considering the high variety in the corpus's sentence lengths, future work could include defining the context window in terms of words instead of sentences and the merging of techniques to reap the benefits of both co-reference resolution and capturing topical information.