Finding the needle in a haystack: Extraction of Informative COVID -19 D anish Tweets (2021)

Abstract: Finding informative COVID-19 posts in a stream of tweets is very useful to monitor health-related updates. Prior work focused on a balanced data setup and on English, but informative tweets are rare, and English is only one of the many languages spoken in the world. In this work, we introduce a new dataset of 5,000 tweets for finding informative COVID-19 tweets for Danish. In contrast to prior work, which balances the label distribution, we model the problem by keeping its natural distribution. We examine how well a simple probabilistic model and a convolutional neural network (CNN) perform on this task. We find a weighted CNN to work well but it is sensitive to embedding and hyperparameter choices. We hope the contributed dataset is a starting point for further work in this direction.

## introduction as of late july 2021, the covid-19 corona virus continues to spread with close to 200m infected patients and more than 4m people who have died to this horrible disease foot_0 striking fear into the hearts of all people around the world. due to this, covid-19 has become the most discussed topic in the news and most of the "official" sources that keep track of the number of dead, infected, and recovered cases, are not frequently kept up to date (e.g. who only updates pandemic information once a day). since the news or online monitoring systems need real-time updates, they turn to social media platforms such as twitter, to gather real-time pandemic information @xcite . however, given that this also happens to be one of the most popular topics on social media, many of these posts contain false or uninformative information @xcite ) that can be difficult to filter out. in 2020, a shared task at wnut @xcite was organized to filter out informative covid-19 related information from an english dataset consisting of 10k covid-19 related tweets. the shared task dataset has close to an equal class distribution between two labels: informative (4,719 tweets) and @xcite . a total of 55 teams submitted different models and it was shown in @xcite that the two best models solved this task with an f1 score of 90.96 using an english covid-twitter-bert model @xcite . in this paper, we keep the real-world non-balanced distribution between informative and non-informative tweets and extend the task to a less-privileged language. to answer the question whether this task is doable in light of the expected skewed distribution, this paper contributes a new dataset for danish by starting from the same annotation guidelines @xcite . due to the high imbalance between classes in this dataset (out of 5,000 tweets, at most 3% are informative tweets), the goal is to create a model that performs well for this task despite the high imbalance. no danish covid twitter bert model exists, hence we experiment with multilingual and danish bert and danish twitter word2vec embeddings. ## dataset this section outlines how the dataset of danish covid-19 related tweets was created, annotated, pre-processed and split for the experiments. ## methodology this section outlines the models that will be used. ## evaluation this section outlines the conducted experiments to examine how well each model performs on the given datasets. ## analysis given that informative-is very infrequent, we analyse the per-class predictions. we examine whether the model learns to accurately predict this class at all. we create a different representation of the confusion matrix, which we call the staircase plot, to better visualize exactly which tweets were labelled correctly and incorrectly by a given model. this enables us to visually compare individual tweet classifications between the different models. an example in figure 1 shows the staircase plot of the best single run produced by the weighted cnn model with static mbert. we notice that the model is very good at informative-, in fact only misses one instance. most confusions stem from the infor-mative+ label, which most often got predicted as uninformative. some of these tweets are in fact difficult for all of the models to classify correctly (see appendix a.4). for example, the sixth informative+ is misclassified by all models as an uninforma-tive tweet. this particular tweet shortly mentions how many people have been put in intensive care due to covid-19 and then questions the need for continuing with the lockdown. we believe that it is the final part of the tweet that confuses the models, possibly also because the word "intensive" has not been weighted high enough in terms of significant feature descriptors of the informative+ class. another common misclassification is on the informative-tweets. the training dataset contains a total number of 15 informative-tweets where 5 of these are in danish and 10 are in english. the imbalance between danish and english in this case is present in the misclassification, since the common tweet that all models get wrong is a danish tweet and the remaining 3 tweets are in english. based on these results it appears that the model learns to distinguish between danish and english and simply predicts english tweets as informative-. we would therefore also expect that english uninformative tweets are misclassified as informative-. this is the case for all models except the cnn mbert model, which uses multilingual embeddings hence having the ability to distinguish between english informative-and uninformative tweets. two challenging uninformative tweets are: one containing some numbers and the word "infected", and the other tweet contains some numbers as well and the word "death". it makes sense that the models would find these tweets difficult given that numbers and the words: "corona", "infect", "recover", "death", and "test", are considered to be strong indicators of an informative tweet when used in the same tweet. ## conclusion we contribute a new danish dataset consisting of covid-19 related tweets and benchmarked neural models. the imbalanced dataset made the task very challenging, but most realistic. the most effective model tested in this paper was the cnn model with class weighting. however, there is no single best embedding setup that consistently outperforms all others on the two development datasets. the cnn model was found to be very sensitive to randomness and hyperparameters, yet loss weighting was essential in this setup as otherwise the cnn performs similar to a simple naive bayes model. future work includes extending the created dataset, comparing supervised training with semi-supervised methods, and a comparison of the cnns used in this work to fine-tuning models.