Winners at W - NUT 2020 Shared Task-3: Leveraging Event Specific and Chunk Span information for Extracting COVID Entities from Tweets (2020)

Abstract: Twitter has acted as an important source of information during disasters and pandemic, especially during the times of COVID-19. In this paper, we describe our system entry for WNUT 2020 Shared Task-3. The task was aimed at automating the extraction of a variety of COVID-19 related events from Twitter, such as individuals who recently contracted the virus, someone with symptoms who were denied testing and believed remedies against the infection. The system consists of separate multi-task models for slot-filling subtasks and sentence-classification subtasks, while leveraging the useful sentence-level information for the corresponding event. The system uses COVID-Twitter-BERT with attention-weighted pooling of candidate slot-chunk features to capture the useful information chunks. The system ranks 1st at the leaderboard with F1 of 0.6598, without using any ensembles or additional datasets.

## introduction the world health organization declared covid-19, a global pandemic on march 11, 2020. as of 2020/09/21, there are over 30 million cases foot_1 and 900,000 deaths due to the infection. with the imposed lockdown, work from home and physical distancing, social media like twitter saw an increased usage. a large part of the use was posting and consuming information on the novel infection. these information include potential reasons for contraction of the disease, such as via exposure to a family member who tested positive, or someone who is showing covid symptoms but was denied testing. accompanying to the pandemic was an infodemic of misinformation about covid-19, including fake remedies, treatments and prevention-suggestions in social media @xcite . @xcite show the possibility to automatically extract structured knowledge on covid-19 events from twitter and released a dataset of covid related tweets across 5 event types. we used this dataset in our experiments for the sharedtask. these tweets are annotated for whether they belong to an event (we refer to this as the event-prediction task in this paper) and their eventspecific questions (factual or opinion). we identify these event-specific questions into two types of subtasks, slot-filling and sentence classification. our system consists of separate multi-task models for slot-filling subtasks and sentenceclassification subtasks. our contribution comprises improvement upon the baseline (mentioned in section 2) in three ways: ## related works sentence classification tasks (such as opinion or sentiment mining) as well as slot-filling tasks have greatly progressed with deep learning advancements such as lstm @xcite , tree-lstm @xcite and transfer learning over pre-trained models @xcite @xcite . among these, ct-bert outperforms others on covid related twitter tasks @xcite . taking inspiration from the same, we use ct-bert as part of our architecture. a variety of slot-filling approaches have been built on top of these deep learning advancements @xcite . the proposed baseline for our task (zong et al., 2020) modifies bert model for slot-filling problem inspired by baldini soares et al. (2019). due to the excellent performance offered by bert (devlin et al., 2019) and baldini soares et al. (2019), we build upon this baseline approach. extraction of structured knowledge from tweets pertaining of events @xcite has been studied for disaster and crises management @xcite and in pandemic scenarios @xcite . extracting such entities can be useful for epidemiologists, deciding policies and preventing spread @xcite . due to the fast-spreading nature of the infection, it is also difficult to manually trace the spread of the pandemic. however, with twitter event-specific entity extraction and geo-location, one could potentially build a real-time pandemic surveillance system @xcite . @xcite show that healthissues related misinformation is prevalent in social media, while @xcite talks about covidspecific misinformation. such systems for extracting structured knowledge over the tweets talking about potential cures for covid will help study how users perceive the covid misinformation. in §3, we describe the dataset and the problem statement. then in §4, we discuss the details of our two multi-task models followed by experiments, results and conclusion. ## dataset and problem statement now, we will briefly go over the dataset. the reader may refer @xcite for full details. each of the 7500 tweets in the dataset belongs to one of the 5 event types: tested-positive, tested-negative, can-not-test, death, and cure. the first four events aimed at extracting structured reports of coronavirus related events, such as self-reported cases or news stories about public figures who were exposed to the virus. each tweet was first annotated for whether it belongs to its respective event (e.g. is the tweet belonging to the tested-positive event talking about someone who tested positive?). throughout this paper, we refer to this as the event-prediction task. the tweets that correspond to its event were then annotated for event-specific questions or sub-event # tweets tasks about factual information and user's opinions. all annotations are done by multiple amazon mechanical turks with inter-annotation agreement. the event-specific questions or subtasks (e.g. name, age, gender of the person tested positive) varies depending on the event. these subtasks are of two categories: slot-filling (e.g., who tested positive/negative?, where are they located?, who is in close contact with person contracting the disease?) and sentence classification (e.g. is author related to infected person?, does the author experience any symptoms?, does the author believe a cure method is effective?). the dataset released tweet ids and their annotations. we obtain our text corresponding to tweets using the official twitter api foot_2 . we now formally describe the two types of eventspecific subtasks: slot-filling subtasks: assume n slot-filling subtasks {s 1 , s 2 ...s n }. we set up each slot-filling subtask s i as a supervised binary classification problem. given the tweet t and the candidate slot s, the model f (t, s) → {0, 1} predicts whether s answers its designated question. we extract a list of event sentence classification slot-filling task tested positive gender, relation who,age,recent-visit,when,where,employer,c.-contact tested negative gender, relation who,age,when,where,duration,close-contact can not test relation, symptoms who,when,where death relation, symptoms who,age,when,where cure opinion what is the cure, who is promoting cure sentence classification subtasks: assume m sentence classification subtasks {c 1 , c 2 ...c m , }. given a sentence classification subtask c i aims to learn a model g(t) → {l 1 , l 2 ...l k }, where t is a tweet and l j is a label. here the number of labels can vary depending on the subtask, for example, gender is labelled with {male, female, others/not specified}, relation with {yes, no}, opinion with {effective, no cure, not effective, no opinion} and so on. all these subtasks are 'supervised' classification problems. the dataset is also annotated with whether a tweet corresponds to its respective event or not. we treat this as an additional event-prediction task. this is a binary classification task that aims to learn a model h(t) → 0, 1 where t is a tweet. ## approach in the following subsections §4.1 and §4.2, we describe our multi-task model for slot-filling and sentence-classification respectively. ## experiments all the experiments were performed using pytorch @xcite and hugging face's transformers @xcite . we use git and wandb @xcite for experiment tracking. optimization is done using adam (kingma and ba, 2014) with a learning rate of 2e-5. slot-filling models are trained for 8 epochs and sentence classification model for 10 epochs. average training time per epoch on tesla p100 is ≈ 4 minutes for slot-filling, and ≈ 30 second for sentence classification. we use a 70-30 split for train-valid set. the valid set is used to obtain the best threshold for each of the slot classification tasks over the grid {0.1, 0.2, ..., 0.9}. we exclude labels with "no consensus" from our data. 5 all the mlp have 1 hidden layer and 0.1 dropout. m lp s j has 4 hidden size, leakyrelu activation @xcite with 0.1 negative slope, rest of the mlp have 50 hidden size and tanh activation. event f1 p r tested positive .68 .80 .58 tested negative .66 .66 .67 can not test .65 .67 .64 death .69 .72 .67 cure/prevention .63 .75 .53 overall .66 .73 .60 ## results our performance on the held-out test set is shown in ## conclusion and future work in this paper, we presented our system that bagged 1st position in the wnut-2020 shared task-3 on extracting covid entities from twitter. we divided the event-specific subtasks into slot-filling and sentence classification subtasks, building separate architectures for the two. for both architectures, we used covid-twitter bert, weightedattention pooling over chunk-spans/sentence and fused logits and features from auxiliary event-prediction task. our ablation studies demonstrated the usefulness of each component in our system. there is a lot of scope of improvement for subtasks with few positive labels. pretraining on relevant data (such as covid-misinformation datasets for event cure) is a promising direction. another direction would be to reduce the training and inference time of slot-filling model by not enclosing the candidate chunk within special @xmath0. we can instead use the attention-weighted pooling over candidate slot chunks. this will reduce the number of bert forward passes from o(k) to o(1), where k is the number of candidate chunks in a tweet.