Target Word Masking for Location Metonymy Resolution (2020)

Abstract: Existing metonymy resolution approaches rely on features extracted from external resources like dictionaries and hand-crafted lexical resources. In this paper, we propose an end-to-end word-level classification approach based only on BERT, without dependencies on taggers, parsers, curated dictionaries of place names, or other external resources. We show that our approach achieves the state-of-the-art on 5 datasets, surpassing conventional BERT models and benchmarks by a large margin. We also show that our approach generalises well to unseen data.

## introduction metonymy is a widespread linguistic phenomenon, in which a thing or concept is referred to by the name of something closely associated with it. it is an instance of figurative language that can be easily understand by humans through association, but is hard for machines to interpret. for example, in they read shakespeare, it is "the works of shakespeare" that we are referring to, not the playwright himself. existing named entity recognition (ner) and word sense disambiguation (wsd) systems have no explicit metonymy detection. this is an issue as named entities and other lexical items are often used metonymically. for instance, germany in the context of germany lost in the semi-final refers to "the national german sports team", different to the context i live in germany which the term is used literally. ner systems generally tag both as location without recognition of the metonymic usage in the first, and wsd systems are tied to word sense inventories and generally don't handle metonyms and other sense extensions well @xcite . intuitively, metonym resolution should improve ner and wsd, something we explore in this paper. metonymy resolution is the task of determining whether a potentially metonymic word ("pmw") in a given context is used metonymically or not. it has been shown to be an important component of many nlp tasks, including machine translation @xcite , question answering @xcite , anaphora resolution @xcite , geographical information retrieval @xcite , and geo-tagging @xcite . conventional approaches to metonymy resolution have made extensive use of taggers, parsers, lexicons, and corpus-derived or hand-crafted features @xcite @xcite . these either rely on nlp pre-processors that potentially introduce errors, or require external domain-specific resources. recently, deep contextualised word embeddings @xcite and pre-trained language models @xcite have been shown to benefit many nlp tasks, and part of our interest in this work is how to best apply these approaches to metonymy resolution. while we include experiments for other types of metonymy, a particular focus of this work is locative metonymy. previous work has suggested that around 13-20% of toponyms are metonymic @xcite @xcite , such as in vancouver welcomes you, where vancouver refers to "the people of vancouver" rather than the literal place. our contributions are as follows. first, we propose a word masking approach, which when paired with fine-tuned bert @xcite , achieves state-of-the-art accuracy over a number of benchmark metonymy datasets: our method outperforms the previous state-of-the-art by 5.1%, 12.2% and 4.8%, on semeval @xcite , relocar @xcite and wimcor (kevin and michael, 2020), respectively, and also outperforms a conventional fine-tuned bert model by a large margin. second, in addition to intrinsic evaluation of location metonymy resolution, we include an extrinsic evaluation, where we incorporate a locative metonymy resolver into a geoparser, and show that it boosts geoparsing performance. third, we demonstrate that our method generalises better cross-domain, while being more data efficient. finally, we conduct a detailed error analysis from the task rather than model perspective. our code is available at: https://github.com/haonan-li/twm-metonymy-resolution . ## related work in early symbolic work, metonymy was treated as a syntactico-semantic violation @xcite . as such, the resolution of metonymy was based on constraint violation, usually based on the selectional preferences of verbs @xcite . @xcite were the first to treat metonymy resolution as a classification task, based on corpus and linguistic analysis. they demonstrated that grammatical roles and syntactic associations are high-utility features, which they subsequently extended to include syntactic head-modifier relations and grammatical roles @xcite . to tackle data sparseness, they further introduce simpler grammatical features by integrating a thesaurus. much of this work has been preserved in recent work in the form of hand-engineered features and external resources. semeval 2007 task 8 on metonymy resolution @xcite further catalyzed interest in the task by releasing a metonymy dataset with syntatic and grammatical annotations, and fine-tuning the task definition and evaluation metrics. a range of learning paradigms (including maximum entropy, decision trees, and naive bayes) were applied to the task. top-ranking systems @xcite @xcite used features provided by the organisers, such as syntactic roles and morphological features. most systems also used features from external resources such as wordnet, framenet, verbnet, and the british national corpus (bnc). later work @xcite @xcite used the wikipedia category network to capture the global context of pmws, to complement local context features. all the above-mentioned approaches resolve metonymy by enriching the information about pmws, in particular via resources. in contrast, our approach is end-to-end: information is contained in the pretrained embeddings and language models only. another difference is that we focus on the context of the pmw only, and not the pmw itself. more recently, in a departure from using ever-more hand-crafted features, @xcite proposed a metonymy resolution approach based on basic parsing features and word embeddings. the main idea is to eliminate words that are superfluous to the task and keep only relevant words, by constructing a "predicate window" from the target word via a syntactic dependency graph. the classification of the target word is then based on the "predicate window". similar to us, they do not take the identity of the target word into consideration. however, we remove the dependency on a dependency parser, and more systematically generate a context representation by masking the target word within a pretrained language model. researchers have released several datasets for metonymy resolution, including semeval @xcite , relocar and conll @xcite , gwn @xcite , and wimcor (kevin and michael, 2020). however, none of them have analyzed the data distribution and or generalisation across datasets. in this paper, we train our model on different datasets, and evaluate its transfer learning abilities. ## approach formally, given a sentence and target word foot_0 contained within it, metonymy resolution is the classification of whether the target word is a metonym or not, and what metonymic readings it has. ## experimental details in this section, we detail the five datasets used in our experiments, and then provide details of the models used in this research. ## results to evaluate metonymy resolution, we train each model over 10 runs and report the average accuracy and standard deviation. for geoparsing, we use precision, recall, and f1-score, based on 5-fold crossvalidation. comparing the different datasets, the relative accuracies vary substantially: semeval loc is the most difficult, while wimcor is the simplest, even with the lexically-split training and test data. with the original data split for wimcor, the result is over 99.5 even with bert-base without masking (and even higher for the other bert-based methods). although it is not the main focus of this paper, we also report the results for the non-locative dataset in we further analysed the attention weights of the different fine-tuned bert models with and without target word masking. we compare the attention weights for each layer separately (12 vs. 24 for bert-base and bert-lg, resp.): we get the attention weight of each head on the target word, and average the heads' weights to generate a single sample point. we found for both models, attention on the target word is substantially higher for the last 4-5 layers, as shown in figure 1 . moreover, target word masking makes the model attend more to the target word. figure 2 is the training curve for the bert models over relocar. we find that, generally, bert-lg converges a bit slower than bert-base, but in each case, the masked model performs substantially better 1 2 3 4 5 6 7 8 epoch 0.750 0.775 0.800 0.825 0.850 0.875 0.900 0.925 0.950 accuracy base lg -mask +mask figure 2: training curve for the bert models over relocar. we show the averaged results for ten runs in each setting, with the shading indicating variance. ## error analysis to further understand the task of metonymy resolution and why the model fails in some cases, we conducted a manual error analysis over a random sample of 150 errors from semeval loc and relocar. we roughly categorise the errors into 6 types, with each instance classified according to a unique error type. some instances had multiple errors among types 4, 5, and 6, in which case we classified them in the priority of 6 > 5 > 4. ## conclusions and future work in this paper, we proposed a word masking approach to metonymy resolution based on pre-trained bert, which substantially outperforms existing methods over a broad range of datasets. we also evaluated the ability of different models in a cross-domain setting, and showed our proposed method to generalise the best. we further demonstrated that an end-to-end metonymy resolution model can improve the performance of a downstream geoparsing task, and conducted a systematic error analysis of our model. the proposed target word masking method can be applied to tasks beyond metonymy resolution. numerous word-level classification tasks lack large-scale, high-quality, balanced datasets. we plan to apply the proposed word masking approach to these tasks to investigate whether it can lead to similar gains over other tasks.