Go Figure! Multi-task transformer-based architecture for metaphor detection using idioms: ETS team in 2020 metaphor shared task (2020)

Abstract: This paper describes the ETS entry to the 2020 Metaphor Detection shared task. Our contribution consists of a sequence of experiments using BERT, starting with a baseline, strengthening it by spell-correcting the TOEFL corpus, followed by a multi-task learning setting, where one of the tasks is the token-level metaphor classification as per the shared task, while the other is meant to provide additional training that we hypothesized to be relevant to the main task. In one case, out-of-domain data manually annotated for metaphor is used for the auxiliary task; in the other case, in-domain data automatically annotated for idioms is used for the auxiliary task. Both multi-task experiments yield promising results.

## introduction we use metaphors in our everyday life as a means of relating our experiences to other subjects and contexts @xcite ; it is commonly used to help us understand the world in a structured way, and oftentimes in an unconscious manner while we speak and write. it sheds light on the unknown using the known, explains the complex using the simple, and helps us to emphasize the relevant aspects of meaning resulting in effective communication. there is a large body of work in the literature that discusses how metaphor has been used in the context of political communication, marketing, mental health, teaching, assessment of english proficiency, among others @xcite @xcite @xcite @xcite @xcite @xcite @xcite ; see chapter 7 in @xcite for a recent review. in the nlp universe, there's been substantial recent interest in automated detection of metaphor @xcite @xcite @xcite @xcite . this paper describes the ets entry to the 2020 metaphor detection shared task held as a part of the 2nd workshop on processing figurative language, at acl 2020 foot_0 . the shared tasks consists of four tracks: all content parts of speech -nouns, verbs, adjectives, and adverbs (allpos) and a verbs-only track (verbs) for two corpora -(a) a corpus of well-edited bnc articles from a variety of genres annotated using the mip-vu protocol, and (b) a corpus of medium to high quality timed, non-native essays written for the test of english as a foreign language annotated under a different protocol. we participated in all the four tracks. our contribution consists of a sequence of experiments using bert, starting with a baseline, then strengthening it by spell-correcting the toefl corpus (section 4). we then devised a multi-task learning setting, where one of the tasks is the token level metaphor classification as per the shared task, while the other is meant to provide additional training that we hypothesized to be relevant to the main task (section 5). the first multitask learning is the utilization out-of-domain data annotated for metaphor, albeit under a different annotation protocol, by using data from the other competition corpus. thus, we use metaphor prediction on the vua corpus as an auxiliary task for the main toefl task, and vice versa. we show that this setup resulted in an improved performance on the toefl test data but not on vua data. a sanity-check experiment where the two training datasets were simply merged together yielded performance that was inferior to the baseline for all tracks (section 5.1). the second auxiliary task is utilization of a large in-domain corpus that we automatically tagged for occurrence of a different type of figurative language phenomenon -idioms. we hypothesize that the affinity between the two ways of figuration might help the system become more sensitive to metaphor by learning to attend to idioms. our results provide support to the hypothesis, as this setting yielded our best result on the vua dataset (section 5.2). we provide a discussion of our findings in section 7. ## baseline system we build our baseline system based on bert @xcite . bert (bidirectional encoder representations from transformers) is a transformer @xcite model that is pretrained on a large quantity of texts, and obtained state-of-the-art performance on many nlp benchmarks @xcite @xcite . since its introduction, there have been many improvements over the original bert model, such as roberta @xcite , ernie @xcite , xlnet @xcite ; we use the most basic model (bert-baseuncased). we fine-tune the bert model as a standard token classification task, that is, after obtaining the contextualized embeddings of a sentence, we apply a linear layer followed by softmax on each token to predict whether it is metaphorical or not. fig 1 shows the architecture of the baseline model. we tune the hyperparameters based on cross-validation on training data. the fold partitions for the vua corpus are the same as the ones used for experiments @xcite . for the toefl corpus, we obtained the folds information from the shared task organizers directly. we select batch size in {16, 32, 64}, number of training epochs in {2, 3, 4, 5}, and use a fixed learning rate of 3 × 10 -5 . we also apply the learning rate scheduler known as slanted triangular @xcite . due to the imbalanced class distribution in our data (see ## experiment 1: spell correction system proper automatic detection of lexically-anchored phenomena in text often depends on availability of correct spelling in the text. the contribution of spelling correction to other tasks has been documented previously, especially for english texts produced by non-native learners of english @xcite . essays written by toefl testtakers are known to contain a considerable amount of spelling errors @xcite . to alleviate this, we used a state-of-the-art automatic spelling corrections system @xcite to correct spelling in the toefl dataset. specifically, for the training partition of the toefl dataset, the system corrected 1553 errors in 180 essays, and 510 errors in 60 essays of the test partition. as we fine-tune bert on relatively small datasets, we attempted to enrich the learning with partially relevant additional materials through a multi-task setting -adding auxiliary tasks and train the metaphor detection task with them. the auxiliary tasks are described in sections 5.1 and 5.2. the model we use for multi-task learning is as follows: instead of directly making predictions based on the output embeddings of bert, the embeddings are first projected to a lower-dimensional representation by a linear layer; each task then has its own classifier on top of that linear layer. the architecture of the model is shown in fig. 2 . during training, the data in batches of the metaphor task (our main task) and the auxiliary tasks are mixed and trained on in an interleaved manner; the specifics will be described for each of the auxiliary tasks separately. in order for the main task to dominate the learning, we also scale the gradient of the auxiliary tasks by a factor of 0.1. the hyperparameters are selected in the same way as described in section 3. ## results tables 4 and 5 show performance of the various systems on allpos and verbs-only tasks, respectively, for both vua and toefl data. since it is clear that spelling correction is useful for improving performance on toefl data, we used the spell corrected version of the data for all the systems from experiments 2 and 3 on toefl data. our best-performing systems reported here are also benchmarked against other participating systems in the shared task summary report @xcite . we obtained a ranking of 2nd and 4th in the vua and toefl tasks, respectively. since vua data contains well-edited bnc text, we did not run spelling correction on vua data. for the verbs tracks, we experimented with both (a) training on allpos data and evaluating on the verbs-only subset of the test data, and (b) training and testing on verbs only subsets. version (a) yielded better results, which are reported here. ## discussion first, we observe that comparative results across the different systems are highly consistent for all-pos and verbs-only settings; we therefore focus on allpos in the discussion. combining the training data from toefl and vua sets does not result in better performance on either test set (see d all in tables 4 , 5 ). this could be due to both out-of-domain nature of the two corpora with respect to each other, to the difference in the guidelines under which the two corpora were annotated, and/or to the difference in the distribution of metaphors vs non-metaphors in the two corpora (see however, when set up as a multi-task system with a shared representation, using data from vua as part of the training process results in better performance on toefl test data, with a 2.6 points f1 score gain for allpos (0.666 vs 0.692 in it is interesting to note that the multi-task version of the setting for using out-of-domain data did not result in improvements on vua test data (f1 score of 0.717 vs 0.715). the drop in recall and increase in precision observed for the d mt model on vua data is consistent with the direction of the results where the two training datasets were simply merged into a bigger training dataset (d all ). it appears that under the guidelines in which toefl data was annotated where argumentation-relevant metaphors are detected intuitively, without recourse to a standard dictionary, the annotation outcomes are more conservative: some instances that the system trained on vua data considered metaphorical were not considered so in a system that was exposed to toefl data bnc toefl 11 find oneself long time other than need-to-know long time pay attention great deal matter of fact once again other than ups and downs day-to-day once more find oneself much less long run come through stay at home old woman play games bear in mind great deal cup of tea jack of all trades day-to-day side effect ask the question much less let alone change one's mind need-to-know ask the question common law again and again close one's eyes well and good blue-eyed tell the truth change one's mind once again we next turn to the experiments with an auxiliary idiom detection task. we observe that on vua data this resulted in a 2-point increase in all pos sys-vua toefl tem p r f p r f bl .721 .713 .718 .701 .563 .624 sp ---.656 .676 .666 d all .728 .676 .701 .576 .637 .605 d mt .741 .692 .715 .669 .717 .692 i mt .721 .749 .734 .718 .616 .663 f1 score -with no penalty in precision, the system gained about 3.5 points in recall (0.713 vs 0.749 on allpos; 0.790 vs 0.823 on verbs). this confirms the usefulness of attending to a related type of figurative language through an auxiliary task -even though the identification of idioms was done using an automated procedure and therefore is quite noisy. to examine the impact of the idiom auxiliary task, we used one of the cross-validation folds as development set. looking at instances tagged as non-metaphor by the baseline model and as metaphor by the current model, there are two observations. first, of the 236 vua sentences with newly tagged metaphors, only 9 sentences contained an idiom, according to our idiom detection system. thus, it does not appear to be the case that it is specifically metaphors within known idioms that the system has now learned to find; this is a tentative conclusion, however, as it is also possible that these sentences did contain idioms that were either not on the list of the 5,000 the system is searching for, or are on the list and are present in the text but are not detected by the system. secondly, it appears that the system has learned some sentence-level characteristics of sentences that contain figurative language, in that quite often multiple words in the same sentence got tagged as metaphors: the 236 sentences contained 323 newly tagged metaphors. the most extreme case is that of 4 new words in the same sentence being tagged as metaphors (italicized): "this desire that can not find its name (though it would dare speak, if it could) is pleasurable." using idioms for an auxiliary task did not help with toefl data. we also tried using the bnc idiom data instead of the toefl 11 idiom data; this resulted in comparable performance, still without improvement over the spell-checked single-task version. since results on vua suggest that idioms could provide useful information for metaphor detection, we intend to further pursue this line of work by attending more closely to the different types of idiomatic expressions that might be more or less useful for metaphor detection, and by improving the idiom detection mechanism. ## conclusion this paper describes the ets entry to the 2020 metaphor detection shared task held as a part of the 2nd workshop on processing figurative language, at acl 2020. we participated in all four tracks -verbs and allpos for each of vua and toefl datasets. our contribution consists of a sequence of experiments using bert, starting with a baseline, then strengthening it by spell-correcting the toefl corpus, followed by a multi-task learning setting, where one of the tasks is the tokenlevel metaphor classification as per the shared task, while the other is meant to provide additional training that we hypothesized to be relevant to the main task. the first multitask learning is the utilization of out-of-domain data annotated for metaphor, albeit under a different annotation protocol, by using data from the other competition corpus; this manipulation helped improve f1 scores for metaphor class on toefl test data, but not on vua data. the second auxiliary task is utilization of a large in-domain corpus that we automatically tagged for occurrence of a different type of figurative language phenomenon -idioms. this manipulation resulted in an improved performance on vua data, but not on toefl data. given the promising results with idiom auxiliary task, we intend to continue work in this direction by improving automatic detection of idioms and by a finer-grained analysis of the contribution of various types of idioms to improve metaphor detection.