Emory at WNUT -2020 Task 2: Combining Pretrained Deep Learning Models and Feature Enrichment for Informative Tweet Identification (2020)

Abstract: This paper describes the system developed by the Emory team for the WNUT-2020 Task 2: “Identifi- cation of Informative COVID-19 English Tweet”. Our system explores three recent Transformer- based deep learning models pretrained on large- scale data to encode documents. Moreover, we developed two feature enrichment methods to en- hance document embeddings by integrating emoji embeddings and syntactic features into deep learn- ing models. Our system achieved F1-score of 0.897 and accuracy of 90.1% on the test set, and ranked in the top-third of all 55 teams.

## introduction until august 31, 2020, the covid-19 outbreak has caused nearly 25 million confirmed cases worldwide, including about 800k deaths @xcite . recently, much attention has been paid to building monitoring systems to track the development of covid-19 by aggregating related data from different sources (e.g., the johns hopkins coronavirus dashboard foot_0 and the who coronavirus disease dashboard foot_1 ). one potentially important source of information is social media, such as twitter and reddit, which provides real-time updates of the covid-19 outbreak. to deal with the massive amount of social media data, several systems have been developed to detect and extract covid-19 related information from twitter @xcite @xcite . however, only a minority of the data collections contain relevant information (e.g., the recovered, suspected, confirmed and death cases as well as location or travel history of the cases) that are useful for monitoring systems, and it is costly to manually identify these informative data. to help address the problem, wnut-2020 task 2 @xcite focuses on attracting research efforts to create text classification systems that can identify whether a covid-19 english tweet is informative or not. this could lead to automated information extraction systems for covid-19, as well as benefit the development of relevant monitoring systems. this paper describes the system developed by the emory team for this task. our solution explores three recent transformer-based models, which are pretrained on large-scale data and achieve great success on different nlp tasks. the pretrained models convert the input document into a embedding matrix, and the first token (i.e., @xcite ) embedding is regarded as the document embedding. we also propose two methods to integrate pretrained deep learning models and empirical features by enriching document embeddings with emoji embeddings and syntactic features. the document embedding is fed into a normalization layer and an output layer, which are fine-tuned with the encoder during the training phase. the output is a probability value from 0 to 1, and the class with the highest probability is chosen. the highest f 1 -score of our system is 0.897, ranking 14 of all 55 teams in the leaderboard foot_2 . ## related work recently, deep learning models pretrained on largescale data, such as bert @xcite , roberta @xcite , and spanbert @xcite , have improved the performance of many downstream nlp tasks such as named entity recognition, semantic role labeling, emotion detection, and question answering. although these models are trained on open domain data such as news articles and english wikipedia, several approaches have been investigated to apply pretrained deep learning models to medical domain tasks. @xcite proposed a dual-context neural model that combines lexical features and personality factors with bert embeddings for suicide risk assessment. @xcite combined bert embeddings and bilstm (bidirectional long short-term memory) to improve the performance of medical text inferences. @xcite applied machine learning models and bert to identify medical domain tweets and reported that bert significantly outperforms some traditional machine learning models such as logistic regression and support vector machines for their task. encouraged by this considerable progress, our work uses recent pretrained deep learning models as document encoders and fine-tunes the classification model on the dataset provided by the task organizers. moreover, we develop two feature enrichment methods to enhance the document embeddings generated by pretrained deep learning models. ## system description figure 1 shows the overall framework of our model for the classification task. in this framework, we use three transformer-based models described in section 3.1 as encoders to generate the document embedding e d , which is the first token (i.e., @xcite ) embedding in our model. the document embedding is normalized by layer normalization and is fed into an output layer with softmax activation. also, we implement two feature enrichment methods to enhance document embeddings by integrating emoji embeddings and syntactic features (described in section 3.2). ## conclusion this paper describes the system developed by the emory team for the wnut-2020 task 2: "identification of informative covid-19 english tweets", including system design, implementation, evaluation, and analysis. we explored three pretrained deep learning models to encode documents, and developed two feature enrichment methods to enhance document embeddings by integrating emoji embeddings and syntactic features. for future work, we will continue to develop our system by exploiting other feature enrichment methods, utilizing external knowledge bases, and investigating other pretrained deep learning models.