Evaluating a B i- LSTM Model for Metaphor Detection in TOEFL Essays (2020)

Abstract: This paper describes systems submitted to the Metaphor Shared Task at the Second Workshop on Figurative Language Processing. In this submission, we replicate the evaluation of the Bi-LSTM model introduced by Gao et al.(2018) on the VUA corpus in a new setting: TOEFL essays written by non-native English speakers. Our results show that Bi-LSTM models outperform feature-rich linear models on this challenging task, which is consistent with prior findings on the VUA dataset. However, the Bi-LSTM models lag behind the best performing systems in the shared task.

## introduction in today's globalized world, text in a given language is not always written by native speakers. it is therefore important to evaluate to what degree nlp models and tools developed and evaluated primarily on edited text written and aimed at native speakers port to non-native language. the metaphor detection shared task at the second workshop on figurative language processing offers the opportunity to perform such an evaluation on a challenging genre: argumentative essays written by non-native speakers of english as part of the test of english as a foreign language (toefl). we participate in the toefl allpos task, a sequence labeling task where each word in running task is labeled with one of two tags: metaphorical (m) or literal (l). while the best-performing system described in this paper was submitted to other sections of the shared task, we focus on reporting a wider range of results for the toefl allpos task. context determines whether a word or phrase is being used in a metaphorical sense. consider an example from the toefl dataset: "the world is a huge stage and nearly everybody is an actor." the words "stage" and "actor" are used metaphorically to analogize the world to a stage and individuals to actors on that stage. a literal usage of these two words would be "the actor walked across the stage.", because "actor" and "stage" both occur within the context of a theatrical performance, which also matches the context of the sentence. beigman @xcite establish baselines for metaphor detection on toefl essays using feature-rich logistic regression classifiers, and show that use of metaphors is a strong predictor of the quality of the essay. the same year, @xcite establish a new state-of-the-art with a simple bi-lstm model on the vua dataset drawn from multiple genres in the british national corpus (bnc). their approach departed from prior models built on linguistically motivated features @xcite @xcite , visual features @xcite or learning custom word embeddings @xcite , and showed that contextualized word representations from bi-lstm can be more effective. in this work, we investigate whether @xcite 's findings can be replicated when detecting metaphors in toefl essays rather than the bnc. in addition, we attempt to answer the following question: do contextualized word representations from a bi-lstm model detect metaphorical word use more accurately than feature-rich linear models? on the one hand, bi-lstm sequence labelers have proven quite successful at learning task-specific representations for many nlp problems. on the other hand, text written by non-native speakers of varying proficiency might include more variability that harms the models ability to learn useful contextual representations. our results show that bi-lstms with word embedding inputs outperform feature-rich linear classifiers as in prior work, but their performances lag behind that of the top performing submissions in the shared task. train test sentences 2741 968 labeled tokens 26647 9014 labeled tokens (+) 1878 -labeled types 5587 2746 the goal of the task is to accurately predict whether words are used in a literal or metaphorical sense in a sequence labeling setting. as shown in in addition to metaphor annotations, the corpus comes with pre-extracted features from @xcite , labeled as provided features in ## official submission our best submission on the leaderboard is a bi-lstm network trained on a spell-checked dataset embedded with glove, elmo, and one-hot unigram lemma vectors. this model yields an f1 score of 0.610, which is slightly below the median score of 0.653. ## conclusion in summary, our experiments replicate existing metaphor detection models in the new settings provided by the toefl allpos task. adding glove vectors and elmo contextual embeddings helped push the performance of the logistic regression model over a simple frequency baseline. the use of a bi-lstm network in combination with glove, elmo, and one-hot unigram lemma vectors yields the highest performance out of all the models tested. this confirms the benefits of contextual representations learned by the bi-lstm for metaphor detection highlighted by @xcite on the vua dataset. however, the more challenging toefl allpos data also shows the limitation of the bi-lstm model, which yields smaller improvements over the baseline than on vua, and lags behind the best systems on the shared task leader board.