GRUBERT : A GRU -Based Method to Fuse BERT Hidden Layers for T witter Sentiment Analysis (2020)

Abstract: In this work, we introduce a GRU-based architecture called GRUBERT that learns to map the different BERT hidden layers to fused embeddings with the aim of achieving high accuracy on the Twitter sentiment analysis task. Tweets are known for their highly diverse language, and by exploiting different linguistic information present across BERT hidden layers, we can capture the full extent of this language at the embedding level. Our method can be easily adapted to other embeddings capturing different linguistic information. We show that our method outperforms well-known heuristics of using BERT (e.g. using only the last layer) and other embeddings such as ELMo. We observe potential label noise resulting from the data acquisition process and employ early stopping as well as a voting classifier to overcome it.

## introduction with the rise of social media, twitter has become an important and oft-used platform for sharing opinions and even doing politics. furthermore, twitter is a rich source of data collection since tweets are often closer to everyday spoken language than formally written texts. this has caused twitter sentiment analysis @xcite to become a well-established benchmark in the scientific community with practical applications such as predicting political preferences @xcite . moreover, it has piqued scientific interest in the field of natural language processing (nlp) as a source of learning informal languages. colloquial language has many unique features as opposed to formal language, such as the use of slang words, misspellings, abbreviations, metaphors, sarcasm and context-dependent changes in meaning. tweets also make extensive use of hashtags. this high volatility results in approaches well-suited to the analysis of formal texts giving sub-par performance on twitter sentiment analysis. the success of word embeddings such as word2vec and glove @xcite is based on their effectiveness in encoding semantic relationships between words. these were the first instances of word embeddings pre-trained on large corpora of text in an unsupervised fashion. however, one major drawback of these representations is that they do not account for the fact that a word can have different meanings based on its context, i.e. polysemy is not modeled. additionally, words not in the dictionary cannot be easily taken into account in such models, which is a problem for tweets because of their frequent use of abbreviations, misspellings, and hashtags not present in the dictionary. following the idea of elmo, more recent works expand unsupervised language models to a much larger scale by training them on large corpora of free text @xcite @xcite . unlike elmo, which uses a multi-layer bi-lstm @xcite , these models are based on a multi-layer transformer architecture @xcite . similarly to recurrent neural networks (rnns) and lstms, transformers are designed to handle sequential data. however, they are entirely based on attention mechanisms @xcite , which allow to train more powerful language models more efficiently. transformer-based models have reached state-of-the-art performance in many nlp tasks @xcite @xcite @xcite . it has been shown that fine-tuning such pre-trained models is effective for many downstream tasks. @xcite . a recent major breakthrough in the realm of nlp was the advent of bert @xcite , which leveraged bi-directional transformers for language representations. one of the main advantages of using bert for twitter sentiment analysis is that it uses sub-tokens instead of a fixed per-word token. this makes it highly suitable for the twitter dataset that often includes misspellings and slang words. moreover, contextualized word representations can be extracted from hidden layers of the bert model @xcite . however, one of the main challenges is the question of how and which layers to use in order to fully optimize the performance for the downstream task @xcite . different layers of the model have been shown to capture different linguistic information @xcite . while earlier layers capture more low level information such as character-based features, the middle layers tend to capture syntactic information and later layers more semantic features @xcite . hence, finding a good way to leverage the relevant information for a specific language task becomes an important problem. another research problem involving microblogging is the limitations rooted in automatic data collection. various ways of automatically labelling twitter data have been developed, such as @xcite which use emoticons to label tweets. these processes inevitably introduce label noise in the training dataset, as is also stated in @xcite . problem setup our training dataset foot_0 consists of 2.5 million labeled tweets, of which one half used to contain a positive smiley ":)" and the other half a negative smiley ":(". those which previously contained a positive smiley are labeled as positive, and those which previously contained a negative smiley are considered to be negative. since emoticons are not a perfect indicator for positivity or negativity of a tweet (especially due to phenomena like sarcasm and irony), we suspect that the dataset potentially contains label noise. given a tweet, our task is to predict its sentiment as either positive or negative. we split the dataset into a 70% training dataset and a 30% validation dataset. furthermore, we use a separate test dataset consisting of 10 thousand unlabeled tweets (excluded from the aforementioned 2.5 million tweets) to report test accuracy. contributions we make the following contributions: ## models and methods in this section, we describe our pipeline for twitter sentiment analysis. foot_1 ## results in this section we first discuss the baselines we compare our models to and report in tables 2 and 3 mean accuracies of three runs for each experiment, as well as the standard deviation. we report the accuracies of the method based on the test split of the dataset. major baselines to assess the usefulness of the learned contextualized representations of our models, we implement a baseline architecture similar to the grubert architecture as follows: after the embedding layer, there is a single bi-gru (analogous to the embedding bi-grus from grubert), followed by another bi-gru (analogous to the downstream bi-gru from grubert), followed by a dense layer with relu activation, dropout, and dense layer with a softmax over the logits as in the grubert architecture. we also chose to compare the intermediate embeddings of our models with two bert hidden layer combination heuristics used in @xcite . all baseline models are trained using the same hyperparameters as in section 2.3, except for elmo, which is trained with a learning rate of 1 • 10 -3 . all baselines use one embedding bi-gru followed by the rest of the downstream architecture, i.e. one embedding bi-gru and a classifier. bert-cat and bert-share we also observe that certain configurations of grubert outperform other commonly used embeddings such as glove, elmo, as well as other common ways of using bert embeddings, such as using only the last layer or concatenating the last four layers. model mean acc. (std. dev.) bert-share-3-linear 89.43 (0.17) bert-share-3 89.66 (0.18) our method can be easily extended to other multi-layer embeddings such as roberta @xcite , leaving possibilities of adapting ensemble methods. we evaluate the final model by implementing our technique on top of a roberta @xcite model and doing an ensemble with various bert-share-3 models trained: (1) as described in section 2, (2) on the full dataset, (3) using multilingual bert embeddings, (4) with a weight decay of 1 • 10 -5 , (5) using roberta embeddings, as well as (6) a bert-share-4 and (7) a bert-share-6 model, both trained with roberta embeddings. using this technique we reach a final test score of 90.94%. ## discussion we show empirically that grubert is superior to standard embeddings for the task of twitter sentiment analysis. however, our model is not easily interpretable and does not allow deeper insights into the bert hidden layers, as is made apparent by the fact that we cannot draw concrete conclusion about why weight sharing gives a boost to certain groupings but not to others. in future work we would like to find interpretable combinations of the bert layers for different task, to better understand the linguistic features present in each hidden layer. as another item of future work, the effect of each component on the overall architecture should be examined more closely. furthermore, the effectiveness of our approach for other nlp tasks remains to be tested. however, our architecture can easily be used as a plugin module for other multi-layer embeddings and downstream models, allowing the effectiveness to be easily examined by future work. ## conclusion we have shown that a dynamic way of combining the bert hidden layers using grus can lead to performance benefits in the case of irregular and plastic language found in tweets. we further experimented with different ways of combining the embeddings and observed that weight sharing can benefit the training process by implicitly inducing regularization and restricting the model complexity. we used early stopping as well as voting classifiers to address the label noise problem inherent in the automatic data collection process using these findings, we develop a framework for the problem of machine-labeled twitter sentiment analysis which makes use of an ensemble of different grubert models to combat label noise.