imec- ETRO - VUB at W - NUT 2020 Shared Task-3: A multilabel BERT -based system for predicting COVID -19 events (2020)

Abstract: In this paper, we present our system designed to address the W-NUT 2020 shared task for COVID-19 Event Extraction from Twitter. To mitigate the noisy nature of the Twitter stream, our system makes use of the COVID-Twitter-BERT (CT-BERT), which is a language model pre-trained on a large corpus of COVID-19 related Twitter messages. Our system is trained on the COVID-19 Twitter Event Corpus and is able to identify relevant text spans that answer pre-defined questions (i.e., slot types) for five COVID-19 related events (i.e., TESTED POSITIVE, TESTED NEGATIVE, CAN-NOT-TEST, DEATH and CURE & PREVENTION). We have experimented with different architectures; our best performing model relies on a multilabel classifier on top of the CT-BERT model that jointly trains all the slot types for a single event. Our experimental results indicate that our Multilabel-CT-BERT system outperforms the baseline methods by 7 percentage points in terms of micro average F1 score. Our model ranked as 4th in the shared task leaderboard.

## introduction covid-19, a highly infectious disease, has dramatically influenced the world. according to official covid-19 related data from world health organization (who), the number of confirmed cases has surpassed 29 million, and the death toll rises to 922,252 as of 15 september 2020 foot_0 . many countries have taken necessary measures, such as social distancing and mask wearing to prevent the spreading of the virus. because of these measures, the communication among people has been changed. as a result, people tend to share their opinions on social media while also obtaining useful information from other users. twitter, a popular social media the goal of the task is to identify that the text spans "tom hanks" and "rita wilson" are slots of type "who", and "australia" is a slot of type "where". the "w-nut 2020 shared task 3: covid-19 event extraction from twitter" is the task of finding useful information from covid-19 related tweets. in this work, we focus on the provided dataset covid-19 twitter event corpus to build a system that can identify text spans (slots) that answer predefined questions related to covid-19 events (e.g., "who is tested positive (negative)?"). this task (i.e., identifying text spans that answer pre-defined questions) has been framed as a slot filling problem by the organizers of the competition and a detailed description of that is available on the work of @xcite . figure 1 illustrates an example of an annotated tweet. that tweet reports a tested positive case. the goal of the task is to identify that the text spans "tom hanks" and "rita wilson" are slots of type "who", and "australia" is a slot of type "where". the rest of the paper is organized as follows. section 2 introduces the related work on slot filling tasks. section 3 describes the dataset we used to build our system along with the tweet preprocessing technique. section 4 presents our pro-posed multilabel-ct-bert system as well as the underlying techniques. section 5 presents the baseline and other methods that have been tested in the context of this competition. section 6 shows our experimental setup and results for the proposed systems and the baseline model. section 7 summarizes our findings and concludes our work on this w-nut shared task. ## related work the slot filling task is the problem where the goal is to identify fine-grained information (related to specific events of interest) from an input sequence. specifically, given a text sequence, the aim is to find relevant text spans for certain types of slots @xcite , where in our case are the different slot types (e.g., "who", "where") for different event types (e.g., tested positive, tested negative). for the slot filling task, several approaches have been proposed in the literature. in particular, convolutional neural networks (cnns) and recurrent neural networks (rnns) have been exploited in the works of @xcite ; @xcite ; @xcite . more recently, @xcite developed a bert-based joint intent classification and slot filling model. their system relies on the pre-trained bert model to encode the input sequences, and jointly trains the intent classification and slot filling tasks by maximizing the conditional probability between the two tasks. in another work, @xcite introduced a slot filling model called span-convert that is based on the convert language model @xcite to extract text spans from dialogs. in this work, we focus on developing a system that can extract covid-19 related information from twitter messages. to do so, we make use of a language model that is optimized for covid-19 related tweets, and we fine-tune the system by jointly training the different slots. ## dataset we use the dataset provided by the shared task for our proposed system. the name of the dataset is @xcite . ## proposed system in this section, we first introduce the underlying methods (bert and ct-bert) of our system. then, we describe the proposed multilabel-ct-bert system in detail. ## other systems in this section, we describe several other systems including the baseline model, the multilabel-bert system, the nli (natural language inference) system and the pairwise system. prediction ct-bert [cls] <@xmath0, and a multilabel classifier on top of the ct-bert model in the grey rectangle. the small yellow and green rectangles represent the input sequence and the hidden representations, respectively. c is the hidden state of the [cls] token. t x is the corresponding hidden state of the input token x. the red rectangle is the prediction of the current enclosed chunk. the input to this system is a tweet text with an enclosed candidate chunk, and the hidden state of the entity @xmath1. ## experiments and results we compare our multilabel-ct-bert system with other systems (i.e., nli, pairwise and baseline) to demonstrate the effectiveness of the proposed architecture. our code is available on github foot_1 . ## conclusion in this paper, a covid-twitter-bert based multilabel-ct-bert system is proposed in the context of the w-nut shared task to deal with the slot filling problem of the recently introduced covid-19 twitter event corpus. our experimental results illustrate that the proposed multilabel-ct-bert system outperforms the baseline and other proposed models in terms of micro avg f 1 and macro avg f 1 scores.