ALBERT - B i LSTM for Sequential Metaphor Detection (2020)

Abstract: In our daily life, metaphor is a common way of expression. To understand the meaning of a metaphor, we should recognize the metaphor words which play important roles. In the metaphor detection task, we design a sequence labeling model based on ALBERT-LSTM-softmax. By applying this model, we carry out a lot of experiments and compare the experimental results with different processing methods, such as with different input sentences and tokens, or the methods with CRF and softmax. Then, some tricks are adopted to improve the experimental results. Finally, our model achieves a 0.707 F1-score for the all POS subtask and a 0.728 F1-score for the verb subtask on the TOEFL dataset.

## introduction as a common rhetorical device, we often use metaphors to express our feelings and ideas vividly and concisely in our daily life. detecting metaphors in texts is of great significance for analyzing the meaning and polarity of sentences. it can also be used to generate sentences that are more suitable for human expression, and promote the development of chat robots, machine translation and other fields. metaphor detection generally recognizes metaphorical words or phrases from the metaphorical sentence, such as "she is a woman with a stone heart", in which "stone" is a metaphorical word modified "heart". however, the task of metaphor detection is very challenging. firstly, metaphor detection is a sequence labeling task, and every word in a sentence needs to be classified. secondly, the boundaries between metaphors and non metaphors are sometimes vague. moreover, due to the different identities of authors, some metaphorical words involve knowledge in specific fields and are difficult to recognize directly @xcite . the traditional lexicon based method cannot cover all possible words occurred in metaphors. it is difficult to recognize metaphors when certain words are outof-vocabulary. although the traditional machine learning method needs to extract features manually @xcite , its performance is still insufficient.while with the further development of language model, different kinds of end-to-end pre-trained models almost dominate the field of natural language processing, and also improve the prediction accuracy of various tasks to a higher level. hence, in this paper we use pre-trained models to deal with metaphor detection task. the purpose of this metaphorical shared task is to identify the whole words and verbs in given sentences. in this paper, we design an albert-bilstm structure to recognize metaphorical words in toefl dataset. firstly, we conduct an experimental comparison on the form of input sentence, and then select the form of inputting the single sentence directly. secondly, we compare the application of bert on this sequence labeling problem, and extract the input form of the first part after the bpe word segmentation of bert. finally, the effect of conditional random field (crf) and softmax with class weights in the output layer is compared and the result shows that softmax with class weights is better. at the same time, we also adopt some tricks in the training process, including semantic merge and loss with class weight. the final result in the test set achieves a 0.707 f1-score for the all pos subtask, and a 0.728 f1-score for the verb subtask on toefl dataset. ## related works at present, researchers in the field of natural language processing have made a lot of effort in metaphor detection task. @xcite used unsupervised learning to detect metaphors, and applied the syntactically perceived distribution word vectors. @xcite used metaphorical language detection as a method to explore the composition of word vectors, and calculated cosine distance to distinguish metaphor from non-metaphor: words that are out of context in sentences may be metaphorical. @xcite proposed a model to connect the expression of glove and elmo for solving the sequence labeling task, which is also transferred to the following metaphor task. @xcite used the flexibility of word vectors to study metaphor and its possibility of modeling in semantic space. @xcite designed an end-to-end model based on glove and elmo, which could identify metaphors conveniently. for metaphor often contains emotions, some researchers tended to carry on emotion analysis on metaphors. @xcite constructed an lexicon based model for analyzing emotions of metaphors. @xcite proposed a new method, which integrated the trigger factors of cognition, emotion and perception. verb metaphor recognition is also an important subtask of metaphor recognition. @xcite used conditional random fields(crf) model and maximum entropy(me) model to recognize verb metaphor, and they pointed out that there were no mature syntactic and semantic tools for metaphor analysis in chinese. beigman @xcite studied the effectiveness of semantic generalization and classification in capturing the rules of verb behavior, and tried to analyze their metaphors from the orthographic words unigrams. these studies also provided some guidance to our work. for example, the word vector concatenation in lstm is similar to rnn_hg @xcite . ## task definition the dataset of this metaphorical shared task includes two kinds: vua @xcite and toefl @xcite . this paper mainly conducts experiments on toefl data. toefl dataset contains 180 articles written by non-native english speakers in the toefl test, and 60 articles in the test set. each article is divided into several parts by sentence. at the same time, the corresponding examination questions of each article are provided, and there are 8 kinds of questions. the details of the dataset are as follows in we make statistics on the sentence length distribution in the data set, and the following is shown by the box chart. it can be seen that the sentence length of training set is longer than that of test set, but most of them are distributed between 0 and 100, no more than 350 tokens. it is suitable for bert model, because the maximum sentence length that bert can support is 512. this shared task subtask is divided into all pos recognition and verb recognition. it also provides some tokens' id in sentences of the test set, and finally submits the recognition result corresponding to token id. final ranking and results are reported by @xcite . ## method in this section, we aim to introduce the method conducted on toefl dataset in this shared task. we use 'bert+bilstm+softmax' as the baseline model, in which bert @xcite is a pretrained language model proposed by google in 2018, and bilstm is a bidirectional recurrent neural network. the details of our method are described below. ## conclusion in this paper, we propose a method with al-bert+bilstm+softmax to identify metaphor words in the sentence. we extract text features through albert's learning ability, and use bil-stm to get contextual representation, then get the final prediction results with softmax layers. we also try several data preprocessing methods and utilize three tricks to improve the performance of our proposed model. besides, we analyze and explain the results of each method according to the characteristics of the metaphor detection task. the experimental results show the effectiveness of our method.