Combining BERT with Static Word Embeddings for Categorizing Social Media (2020)

Abstract: Pre-trained neural language models (LMs) have achieved impressive results in various natural language processing tasks, across different languages. Surprisingly, this extends to the social media genre, despite the fact that social media often has very different characteristics from the language that LMs have seen during training. A particularly striking example is the performance of AraBERT, an LM for the Arabic language, which is successful in categorizing social media posts in Arabic dialects, despite only having been trained on Modern Standard Arabic. Our hypothesis in this paper is that the performance of LMs for social media can nonetheless be improved by incorporating static word vectors that have been specifically trained on social media. We show that a simple method for incorporating such word vectors is indeed successful in several Arabic and English benchmarks. Curiously, however, we also find that similar improvements are possible with word vectors that have been trained on traditional text sources (e.g. Wikipedia).

## introduction social media has become an important source of information across numerous disciplines @xcite . for instance, it allows extracting and analyzing people's opinions, emotions and attitudes towards particular subjects, in a way which is difficult to achieve using other information sources. however, social media posts tend to be short and often contain abbreviations, slang words, misspellings, emoticons and dialect @xcite . for language models (lms) such as bert @xcite , which have been primarily trained on wikipedia, this poses a number of clear challenges. in the case of arabic, the challenge is even greater, since social media posts are mostly written in regional dialects, which can be different from the language that is found in resources such as wikipedia @xcite . in particular, the arabic language can be divided into classical arabic, modern standard arabic, and dialectal arabic @xcite . the latter differs between arabic countries, and sometimes among regions and cities. social media acts as the primary source where arabic dialects appear as written text, due to the informality of these platforms. similar as for english, the best results in many arabic nlp tasks are currently obtained with lms. in particular, the arabert model (antoun et al., 2020) has achieved state-of-the-art results in sentiment analysis, named entity recognition and question answering, among others. however, arabert was trained on wikipedia and news stories. it has thus not seen the arabic dialects in which most social media posts are written. surprisingly, however, @xcite found that arabert is nonetheless able to outperform other methods on social media tasks. this includes methods that use the aravec @xcite embeddings, which are word2vec vectors trained on twitter, and have a wide coverage of dialect words. our hypothesis is that arabert and aravec have complementary strengths, and that better results can thus be obtained by combining these two resources. similarly, for english tasks, we would expect that the performance of bert on social media can be improved by incorporating word embeddings that have been trained on social media. however, for english we would expect to see a smaller effect, since compared to arabic, the vocabulary of english social media is more similar to the vocabulary in traditional sources. to test these hypotheses, we propose and evaluate a simple classifier which combines language models with static word embeddings. our main findings are that incorporating word vectors can indeed boost performance. surprisingly, this even holds for word embeddings that have been trained on standard sources. ## related work while there is a large literature on nlp for social media, more efforts that focus on the arabic language are needed. a notable work is @xcite , which developed a cnn and lstm ensemble model for arabic sentiment analysis. they used aravec pre-trained word embeddings for the word embedding representation. recently, @xcite proposed an approach that relies on the concatenation of pre-trained aravec and fasttext vectors. however, the best results on most datasets are currently achieved by fine-tuning arabert @xcite , as already mentioned in the introduction. for the english language, nguyen et al. (2020) recently introduced bertweet, a bertbased language model that was trained on a large corpus of english tweets. their experiments show that utilising this model led to improved results on different tasks involving twitter posts, such as named entity recognition, part-of-speech tagging and text classification. in this work, we investigate the effectiveness of combining pre-trained language models with static word embeddings. for earlier language models, most notably elmo @xcite , it was common practice to combine contextual embeddings, predicted by the language model, with static word embeddings. however, the introduction of bert has essentially eliminated the need for static word vectors in standard settings. on the other hand, several authors have shown that it can be beneficial to incorporate entity vectors with bert, allowing the model to exploit factual or commonsense knowledge from structured sources @xcite . ## proposed approach there are various ways in which bert-based models can be combined with static word vectors. note, however, that we cannot simply concatenate the contextualised word vectors predicted by bert with the corresponding static word vectors, due to the fact that the tokenization strategy used by bert means that many words are split into two or more word-piece tokens. one possible solution, adopted by @xcite in a different setting, is to combine the word-piece tokens from the same word into a single vector, using a convolutional or recurrent neural network. the resulting word-level vector can then be concatenated with the corresponding static word vector. however, without a large training set, there is a risk that the representations predicted by bert are degraded by this aggregation step. as a simpler solution, we instead combine representations obtained from bert and from the static word vectors at sentence level. in particular, to obtain a sentence vector from the fine-tuned bert model, we simply take the average of the predicted contextualised vectors. to obtain a sentence vector from the static word embeddings, we use either a convolutional neural network (cnn) or a long short term memory network (lstm). after concatenating the two types of sentence vectors, we apply dropout, followed by a softmax classification layer. a diagram illustrating the model is shown in figure 1 . rather than jointly training the combined model, we first fine-tune the bert model on its own. after this fine-tuning step, we freeze the bert model and train the cnn and combined classification layer. we found this strategy to be more robust against over-fitting. ## experimental results we experimentally analyze the benefit of incorporating static word vectors, in both arabic and english. for arabic, we used the following datasets: ajgt arabic jordanian general tweets (alomari et al., 2017) is a sentiment classification dataset. it consists of 1,800 tweets annotated with positive and negative labels. semeval-2017 task 4 @xcite we use the arabic subtask a dataset from semeval 2017 task 4. this dataset contains 10,126 tweets, annotated with negative, neutral, and positive labels. l-hsab a dataset for hate speech and abusive language in arabic levantine dialect @xcite . it consists of 5846 tweets, which are annotated as normal, abusive or hate. arsentd-lev an arabic levantine dataset for sentiment analysis that discusses multiple topics @xcite . it contains 4000 tweets, labelled as a very negative, negative, neutral, positive, or very positive. for english, we used the following datasets: irony detection we use semeval-2018 task 3 dataset @xcite , containing 4,618 tweets that are annotated as ironic or not. semeval-2019 task 6: offenseval an offensive language identification dataset for social media @xcite . it consist of 14,100 tweets that are labeled as offensive or not. ## conclusions in this paper, we have presented a simple approach to combine static word embeddings with bertbased language models. intuitively, the reason why this hybrid approach can outperform the bertbased models themselves is because the latter were not trained on wikipedia. the alternative solution would be to train language models on a relevant social media corpus, as in the bertweet model @xcite . while such a strategy is likely to lead to a better overall performance, in principle, this is not always feasible in practice. for instance, using static word vectors could play an important role in dealing with emerging terms, such as trending hashtags, as continuously updating language models (for many different languages) would be too expensive. similarly, incorporating static word vectors seems to be a promising strategy for improving language models for low-resource languages, as specialized language models (e.g. trained on social media) are unlikely to become available for such languages.