daminglu123 at S em E val-2022 Task 2: Using BERT and LSTM to Do Text Classification (2022)

Abstract: Multiword expressions (MWEs) or idiomaticity are common phenomenon in natural languages. Current pre-trained language models cannot effectively capture the meaning of these MWEs. The reason is that two normal words, after combining together, could have an abruptly different meaning than the compositionality of the meanings of each word, whereas pre-trained language models reply on words compositionality. We proposed an improved method of adding an LSTM layer to the BERT model in order to get better results on a text classification task (Subtask A). Our result is slightly better than the baseline. We also tried adding TextCNN to BERT and adding both LSTM and TextCNN to BERT. We find that adding only LSTM gives the best performance.

## introduction machine learning has made deep impacts on various areas, such as computer vision @xcite @xcite , computational biology @xcite @xcite , and natural language processing @xcite @xcite . in natural language processing, large pre-trained models are prevailing and have achieved great successes. models such as bert @xcite , roberta @xcite , xlnet @xcite , albert @xcite , ernie @xcite , etc. performed pretty well in tasks such as sentiment analysis, commonsense reasoning @xcite , qa system @xcite and many other tasks. however, these models are not good at certain tasks such as assessing humor and capturing idiomaticity. this shortcoming is largely due to natural languages' flexibility. in this paper, we focus on how to use large pretrained language models to determine whether a multiword expression (mwe) has a trivial meaning @xcite , a.k.a, the compositionality of each word's meaning, or it is an idiomatic usage. we use the dataset provided in @xcite . in the training set, the target mwe is given. the previous sentence, the target sentence and the next sentence are also given. we need to decide if the mwe has an idiomatic meaning or its meaning is trivial. this task then can be treated as a text classification problem. the rest part of this paper is organized as follows: ## dataset and task as mentioned in @xcite , the dataset for subtask a consists of naturally occurring (target) sentences, previous sentences and next sentences. the target sentence contains potentially idiomatic mwes annotated with a finegrained set of meanings: compositional meaning and idiomatic meaning(s). ## methods our core pre-trained language model is mbert @xcite . we chose mbert over bert hoping that it could better fit the task's multilanguage specification. in traditional methods, ngram was used to detect and group the mwes. in our methods, we tried to use either lstm @xcite or textcnn @xcite to capture the mwes. we concatenate lstm or textcnn to mbert in order to increase the performance. ## results we use the mbert with 12 hidden layers. we did experiments on dropouts with 0.1 and 0.2. as mentioned in section 3, we explored of adding either a lstm or a cnn to the final fully connected layer of the transformer from mbert. ## discussion one reason that our method does not boost the performance a lot might be that we add the lstm or textcnn to the end, whose effect is limited to the whole pipeline. another new method, according to @xcite , is that we can turn this classification problem into a masked word problem. in prompt, it claims the integration is more genuine, but choosing the prompt could be technical. another important reason is overfitting. we tried to increase dropout from 0.1 to 0.2 in order to get rid of overfitting. but the effect was opposite. according to @xcite , adding lstm could boost question answering tasks, whereas our task is in fact a text classification. this might be the reason of the tiny improvement.