In December 2019, a novel coronavirus, SARS-CoV-2, emerged and began rapidly spreading around the world. In early 2020, many countries initiated social distancing measures and lockdowns to help slow the spread of the virus which has infected 5.5 million people and led to 347,864 deaths as of 2020/05/26. 1 Because COVID-19 is a novel infectious disease, much is unknown about when and where it is safe to resume various activities. Official data on COVID-19, from agencies such as the World Health Organization (WHO) or Centers for Disease Control and Prevention (CDC) present aggregated views, such as the number of cases per city or county. However, these datasets do not capture statistics about more specific events that might be of interest, for instance people who were refused access to testing. On the other hand, people share a wide variety of information related to COVID-19 publicly on social media. For example, Twitter users often indicate when they might be at increased risk of COVID-19 due to a coworker or 1 https://www.nytimes.com/interactive/ 2020/world/coronavirus-maps.html other close contact testing positive for the virus, or when they have symptoms but were denied access to testing. Moreover, news reports on public figures who are tested positive are also widely circulated on social media, such as British PM Boris Johnson and U.S. Senator Amy Klobuchar's husband.
In this paper, we show that it is possible to automatically extract structured knowledge on events related to COVID-19 from Twitter. We annotate a corpus of tweets describing positive/negative tests, denied access to testing, deaths due to etc . For each tweet that mentions one or more such events, we further annotate text spans within the tweet to fill a set of pre-defined slots that answer important questions (e.g., Who tested positive? Where did they recently travel? Who is their employer?)
There are a number of important factors to consider when using COVID-related information from Twitter. Facts should be independently verified before they are assumed to be true. However, we believe that event extraction may be useful for epidemiologists, journalists or policymakers, helping them find relevant pieces of information. Furthermore, our tools could enable new kinds of aggregate data analysis, which could prove useful so long as consumers of this information clearly understand its limitations. We will make our data available to the research community. To protect users' privacy we will only distribute IDs of the tweets in our dataset. To extract structured knowledge from tweets about COVID-19, we formulate the problem as a slot filling task: given a tweet, first identify whether it corresponds to a relevant event, then find relevant text spans corresponding to a pre-defined list of slots associated with the event type (Benson et al., 2011) . As an example, consider the tweet that reports a CAN-NOT-TEST event in Figure 1 . In this case, the model should fill the LOCATION slot with "Houston" and also identify that the author of the tweet is currently experiencing COVID-19 symptoms. Table 1 shows the overall statistics of our corpus.
Data Collection. We consider five event types related to COVID-19: TESTED-POSITIVE, TESTED-NEGATIVE, CAN-NOT-TEST, DEATH, CURE & PRE-VENTION. 2 The first four events aim to extract structured reports of coronavirus related events, many of which are self-reported cases or come from news stories about public figures who have been exposed to the virus. As there is no widely accepted antiviral treatment or vaccine for COVID-19 as of 2020/06/01, we also dedicate one event type for CURE & PREVENTION to study how some potentially misleading information are perceived by public. We have been continuously collecting our data since 2020/1/15 by tracking keywords using the Twitter API. 3 For example, "tested positive" is used as one of the keywords for TESTED-POSITIVE events. We have annotated 7,500 tweets sampled from 2020/01/15 and 2020/04/26. For duplicate tweets, we only keep the earliest post. We also use Jaccard similarity with a threshold of 0.7 to remove near-identical tweets that are posted same-day. 2 We are actively expanding the list to include more event types, such as WAITING-FOR-THE-TEST-RESULTS, HOSPI-TALIZATION, ICU, RECOVERY, and REOPENING. 3 We list all keywords for data collection in Appendix A.
We use crowd workers on Amazon Mechanical Turk to annotate our dataset. We provide annotation interface in Appendix B. Each tweet is annotated by 7 workers in two steps:
1. Specific Events. Some tweets are generic news reports, however, we are more interested in capturing tweets with fine-grained information, especially the self-reporting cases. Therefore, we first ask the annotators to decide whether a tweet is regarding an individual or a group of people. 4 Annotators proceed to the next step only if they answer yes to this question. 2. Slot Filling. In this second step, we ask questions to collect factual information and user's opinions for filling our designated slots (Table  2) . We provide the annotators with a list of candidate answers, which include all noun chunks and name entities extracted by a Twitter tagging tool (Ritter et al., 2011) . We also combine noun chunks if they are adjacent or separated by a preposition phrase. 5 We include "author of the tweet" as an additional option for who questions and "near author of tweet" for where questions. For each tweet, annotators have an average of 10 to 11 possible answers to choose from, and are allowed to choose more than one answers for WH-questions. We also collect Wikipedia links for public figures involved in the events.
Inter-annotator Agreement. During annotation, we track crowd workers' performances by comparing their annotations with the majority votes and remove workers whose F1 scores fall below 0.65. In addition, we independently annotated 100 tweets (20 tweets for each event) and compared them with the majority votes of crowdsourcing workers. We observe an average of 0.76 Cohen's Kappa (Artstein and Poesio, 2008) for specific-event questions and an average F1 score of 0.74 for slot-filling. given the tweet t and the candidate slot s, the classification model f (t, s) → {0, 1} predicts whether s answers its designated question. To keep the evaluation consistent for all Ts, we convert the nonbinary slot "gender" (with labels M, F, NA) into two independent binary classification tasks.
LR model. We use a logistic regression classifier as a baseline, using n-gram (n = 1, 2, 3) features. We replace the slot s in the tweet t with a special token <TARGET> before computing the n-grams. BERT model. We fine-tune a BERT (Devlin et al., 2019) based classifier that takes a tweet t as input and encloses the candidate slot s, within the tweet, inside special entity start <E> and end </E> markers. The BERT hidden representation of token <E> is then processed through a fully connected layer with softmax activation to make the binary prediction for a task T (Baldini Soares et al., 2019). By design, many slots within an event are semantically related to each other. For example, "gender" slot is directly related to the "who" slot. To leverage this, we jointly train the final softmax layers of all the Ts in an event by sharing their BERT model parameters. We use BERT base cased model and HuggingFace PyTorch implementation (Wolf et al., 2019) . All the shared BERT models are fine-tuned with 2 × 10 −5 learning rate using Adam (Kingma and Ba, 2015) algorithm for 8 epochs. This model has ≈ 110M parameters.
For all the slots and events, the gold labels are assigned based on the annotators' majority votes. We use a 60/15/25 ratio for train/dev/test. The best threshold for each T is found by grid search on dev data. 6 We report the model performance by precision (P), recall (R) and F1 score in Table 3 . 7 The BERT-based classifier significantly outperforms logistic regression on all the slot filling tasks. By exploiting the shared BERT representations across the slots of the same event, the BERT model is able to get reasonably good F1 scores, even on slots that have a limited amount of labeled training data.
Our annotations can be potentially used for other analytical purposes besides event slot filling.
Tracking Self-reporting Cases. One feature of our dataset is that it enables tracking of selfreporting cases. We consider tweets with "who" slot annotated with "I", "we" or "author of the tweet" as self-reporting cases. Table 4 presents the number of such cases for each event. We note a much higher percentage of users tends to report their CAN NOT TEST situations online. We can also automatically detect self-reported cases by training a binary classifier using our annotated corpus. A simple logistic regression with bigram and unigram features can achieve a 0.64 F1 score based on 5-fold cross validation on CAN-NOT-TEST data. et al., 2015) . Perceived Cures and Treatments. There are 219 different descriptions of cure or prevention methods for COVID-19 in our corpus, including widely discussed ones "toilet paper" and "alcohol". The top 5 cure or prevention methods that Twitter users believe effective are "social distancing", "vitamin c", "hydroxychloroquine", "chloroquine" and "bathroom hygiene".
In this paper, we present an annotated corpus of 7,500 tweets for COVID-19 events, including positive/negative tests and denied access to testing. We demonstrated that our corpus supports accurately identifying COVID-19 events in Twitter. We further demonstrated that we are able to extract text spans for filling a set of pre-defined slots specific to each event. Finally we showed that our annotated corpus could be used for analyzing self-reporting cases and user demographics. 
A portion of our annotation interface is shown in Figure A. 1. 
The BERT model for each T is trained on a single GeForce RTX 2080 Ti GPU with average training time per epoch ≈ 2 minutes.
We provide some examples of our annotated tweets for each category in Tables A.2-A.6. We mark a tweet's slot filling annotation as "Not Applicable" if annotators do not think it describes a specific event (discussed in Section 2.1). Please refer to Table 2 for complete slot filling questions.
Event Type: TESTED POSITIVE
My wife's grandmother tested positive for coronavirus in an old persons home in CZ. 9 others tested positive. afaik 1 died. After 1 death they tested all residents and staff. Residents confined to rooms. They got extra staff. The grandmother has recovered now. Why is UK so bad? 
apparently the staff of brikama hospital is now in isolation because a nurse tested positive for covid19...
These numbers are being fluffed. They count everyone who tested positive not necessarily actually that was the cause of death. Asymptomatic people could've died for an unrelated reason and they tally it as a covid19 death. And they are saying asymptomatic people are very common. 