An Empirical Survey of Unsupervised Text Representation Methods on T witter Data (2020)

Abstract: The field of NLP has seen unprecedented achievements in recent years. Most notably, with the advent of large-scale pre-trained Transformer-based language models, such as BERT, there has been a noticeable improvement in text representation. It is, however, unclear whether these improvements translate to noisy user-generated text, such as tweets. In this paper, we present an experimental survey of a wide range of well-known text representation techniques for the task of text clustering on noisy Twitter data. Our results indicate that the more advanced models do not necessarily work best on tweets and that more exploration in this area is needed.

## introduction recent years have witnessed an exponential increase in the usage of social media platforms. these platforms have become an important part of politics, business, entertainment, and general social life. correspondingly, the amount of data generated by users on these platforms has also grown exponentially. though data on social media includes various modalities, such as images, videos, and graphs, text is by far the largest type of data generated by users. thus, in order to extract knowledge and insight from social media, sophisticated text processing models are needed. luckily, in parallel to the growth of social media, there has been a rapid rise in the development of sophisticated text representation techniques, the most recent being large-scale pre-trained language models that use transformer-based architecture @xcite (such as bert @xcite , and xlnet @xcite ). these methods can generate general-purpose vector representations of documents that can be used for any downstream task (e.g., sentiment classification). however, the representation power of these methods for data from social media is not well understood. this is especially true for tweets which are usually short, noisy, and idiosyncratic. this paper is an attempt to evaluate and catalogue the representation power of a wide range of methods for tweets, starting from very simple bag-ofwords representations (or embeddings) to representations generated by recent transformer-based models, such as bert. since we are interested in the general representation power of the methods and not their performance on any specific downstream tasks, we do not fine-tune any of the methods using downstream tasks and use unsupervised evaluation (i.e., clustering) for our survey. ## text representation methods in this section, we briefly introduce the methods used in our survey, sorted from oldest to newest. for word embedding methods like word2vec, glove, and fasttext, which dot not explicitly support sentence embeddings, we average the word embeddings to get sentence embeddings. for deep models like elmo, bert, albert, and xlnet, we take the average of the hidden state of the last layer on the input sequence axis. note that some other works use the hidden state of the first token ([cls]), but in our experiments, we use the pretrained model without fine-tuning, in this case, the hidden state of @xcite is not a good sentence representation. note that we use all these deep neural models without fine-tuning. this is because finetuning is usually based on specific downstream tasks which bias the information in the hidden states, weakening the general representation. note that when we refer to n-gram models we mean models that capture all grams up to and including the n-gram (e.g., bigram models will include bigrams and unigrams). ## experiments since we are interested in measuring the general text representation power of our methods, we use clustering as a way to evaluate the representations generated by each model (instead of any downstream supervised tasks). we use the vector representations of each tweet to run k-means clustering for different values of k. we use two tweet datasets for our evaluation. the tweets in these datasets have labels corresponding to their topic which we use as cluster ground-truth for evaluation purposes. dataset 1 @xcite : this dataset includes 356,782 tweets belonging to 1,036 topics. ## results & discussion for each dataset, we average the scores from k-means clustering with different values of k. though we use several metrics in our evaluations for the sake of being thorough, most of the metrics are in fact highly correlated. fig. 1 shows the correlation between each pair of metrics (calculated based on the clustering results of our methods). we can see that all the external evaluation metrics (homogeneity, completeness, v-measure, ami, and ari, which need external ground-truth labels) highly agree with each other while the internal evaluation metric (silhouette score, which does not need external ground-truth labels) does not. the clustering results are shown in fig. 2 and fig. 3 , the methods in both figures are sorted based on the date of their release to capture the advancements in nlp. unlike conventional tasks and datasets (such as the glue benchmark @xcite ), there does not seem to be a very clear trend of improvement for capturing tweet representations. the more advanced models are not necessarily the best. notably, the bert family of large-scale pre-trained language models (albert, sentence-bert, etc) do not vastly or consistently outperform much simpler methods such as bagof-words and tf-idf. xlnet, on the other hand, seems to be the best performing method for cap- very recently, a large-scale pre-trained bert model for english tweets was trained and released @xcite . this model was released just days before the publication of this paper and thus we did not have time to thoroughly compare its performance against the other models. however, we believe this model is a step in the right direction as we have shown in this paper that models trained on standard english corpora do not perform well on tweets. ## conclusion in this paper, we presented an experimental survey of 14 methods for representing noisy usergenerated text prevalent in tweets. these methods ranged from very simple bag-of-words representations to complex pre-trained language models with millions of parameters. through clustering experiments, we showed that the advances in nlp do not necessarily translate to better representation of tweet data. we believe more work is needed to better understand and potentially improve the performance of the more recent methods, such as bert, on noisy, user-generated data.