I nfo M iner at WNUT -2020 Task 2: Transformer-based Covid-19 Informative Tweet Extraction (2020)

Abstract: Identifying informative tweets is an important step when building information extraction systems based on social media. WNUT-2020 Task 2 was organised to recognise informative tweets from noise tweets. In this paper, we present our approach to tackle the task objective using transformers. Overall, our approach achieves 10th place in the final rankings scoring 0.9004 F1 score for the test set.

## introduction by 31st august 2020, coronavirus covid-19 is affecting 213 countries around the world infecting more than 25 million people and killing more than 800,000. recently, much attention has been given to build monitoring systems to track the outbreaks of the virus. however, due to the fact that most of the official news sources update the outbreak information only once or twice a day, these monitoring tools have begun to use social media as the medium to get information. there is a massive amount of data on social networks, e.g. about 4 millions of covid-19 english tweets daily on the twitter platform. however, majority of these tweets are uninformative. thus it is important to be able to select the informative ones for downstream applications. since the manual approaches to identify the informative tweets require significant human efforts, an automated technique to identify the informative tweets will be invaluable to the community. the objective of this shared task is to automatically identify whether a covid-19 english tweet is informative or not. such informative tweets provide information about recovered, suspected, confirmed and death cases as well as location or travel history of the cases. the participants of the shared task were required to provide predictions for the test set provided by the organisers whether a tweet is informative or not. our team used recently released transformers to tackle the problem. despite achieving 10 th place out of 55 participants and getting high evaluation score, our approach is simple and efficient. in this paper we mainly present our approach that we used in this task. we also provide important resources to the community: the code, and the trained classification models will be freely available to everyone interested in working on identifying informative tweets using the same methodology 1 . ## related work in the last few years, there have been several studies published on the application of computational methods in order to identify informative contents from tweets. most of the earlier methods were based on traditional machine learning models like logistic regression and support vector machines with heavy feature engineering. @xcite investigate tweet newsworthiness classification using features representing the message, user, topic and the propagation of messages. others use features based on social influence, information propagation, syntactic and combinations of local linguistic features as well as user history and user opinion to select informative tweets @xcite @xcite . due to the fact that training set preparation is difficult when it comes informative tweet identification, several studies suggested unsupervised methods. @xcite built a news processing system, called twitterstand using an unsupervised approach to classify tweets collected from pre-determined users who frequently post news about events. even though these traditional approaches have provided good results, they 1 the github repository is publicly available on https://github.com/hhansi/ informative-tweet-identification are no longer the state of the art. considering the recent research, there was a tendency to use deep learning-based methods to identify informative tweets since they performed better than traditional machine learning-based methods. to mention few, alrashdi and o' @xcite suggested an approach based on bidirectional long short-term memory (bi-lstm) models trained using word embeddings. another research proposed a deep multi-modal neural network based on images and text in tweets to recognise informative tweets @xcite . among the different neural network models available, transformer models received a huge success in the area of natural language processing (nlp) recently. since the release of bert @xcite , transformer models gained a wide attention of the community and they were successfully applied for wide range of tasks including tweet classification tasks such as offensive tweet identification @xcite and topic identification @xcite . but we could not find any previous work on transformers for informative tweet classification. hence, we decided to use transformer for our approach and this study will be important to the community. ## methodology the motivation behind our methodology is the recent success that the transformers had in wide range of nlp tasks like language generation @xcite , sequence classification @xcite @xcite , word similarity @xcite , named entity recognition @xcite and question and answering @xcite . the main idea of the methodology is that we train a classification model with several transformer models in-order to identify informative tweets. ## experiments and results in this section, we report the experiments we conducted and their results. as informed by task organisers, we used precision, recall and f1 score calculated for informative class to measure the model performance. results in sections 5.1 -5.3 are computed on validation data set and results in section 5.4 are computed on test data set. ## conclusion we have presented the system by infominer team for wnut-2020 task 2. for this task, we have shown that the ct-bert is the most successful transformer model from several transformer models we experimented. furthermore, we presented several fine tuning strategies: self-ensemble, entity integration and language modelling that can improve the results. overall, our approach is simple but can be considered as effective since it achieved 10 th place in the leader-board. as a future direction of this research, we hope to analyse the impact by different classification heads such as lstm and convolution neural network (cnn) in addition to softmax classifier on performance. also, we hope to incorporate meta information-based features like number of retweets and likes with currently used textual features to involve social aspect for informative tweet identification.