HLTRI at W - NUT 2020 Shared Task-3: COVID -19 Event Extraction from T witter Using Multi-Task Hopfield Pooling (2020)

Abstract: Extracting structured knowledge involving self-reported events related to the COVID-19 pandemic from Twitter has the potential to inform surveillance systems that play a critical role in public health. The event extraction challenge presented by the W-NUT 2020 Shared Task 3 focused on the identification of five types of events relevant to the COVID-19 pandemic and their respective set of pre-defined slots encoding demographic, epidemiological, clinical as well as spatial, temporal or subjective knowledge. Our participation in the challenge led to the design of a neural architecture for jointly identifying all Event Slots expressed in a tweet relevant to an event of interest. This architecture uses COVID-Twitter-BERT as the pre-trained language model. In addition, to learn text span embeddings for each Event Slot, we relied on a special case of Hopfield Networks, namely Hopfield pooling. The results of the shared task evaluation indicate that our system performs best when it is trained on a larger dataset, while it remains competitive when training on smaller datasets.

## introduction with the outbreak of the covid-19 pandemic, people turned to social media platforms, such as twitter, to read and to share timely information about their experiences with testing, treatment and deaths caused by the virus. extracting information about these types of events has the potential to inform covid-19 surveillance systems, which play a critical role in the public health mission of agencies at the international, national and local level. the shared task organized by the 6-th workshop on noisy user-generated text (w-nut) in 2020 focused on extracting covid-19 events from twitter by targeting five types of events of interest, which are illustrated in interestingly, events of type tested positive, tested negative, can not test and death have slots answering questions about when and where, which help ground the events temporally and spatially, potentially informing systems that try to capture automatically trends of testing results for the covid-19 virus. some slots for events of type tested positive, tested negative, also encode answers about demographic information, e.g. age, gender as well as epidemiological information, e.g. close contact or relation. only events of the type tested positive have a slot for employer. some clinical information is available through the slot symptoms of events of type can not test or the slots what cure and who cure of the events of type cure. for the cure type of events, a slot capturing the opinion of the tweet reporter was also annotated, providing a means for analyzing the change in opinions throughout time. the organizers of the w-nut shared task 3 provided participants with a set of 7, 500 annotated tweets, 7, 013 of which we were able to download. the tweets are categorized by the type of event they mention. in each category of tweets, every tweet was annotated with ( 1 ) the text spans that can be mapped in any of the slots corresponding to the event type, as well as (2) the corresponding event slot category. moreover, in each tweet, text spans that were not mapped to any event slot were also provided. the training data we have used contains 3, 794 event type mentions and 10, 778 event slot instances. this timely and richly-annotated twitter dataset allowed participants to design and train their event extraction systems, expecting to be tested on a different set of tweets, which were labeled with the event type, probably a byproduct of tweet retrieval using keywords proven to return relevant posting for the events of interest. in the test set, the tweet text spans are also provided. the challenge of event extraction from twitter texts was defined as an event slot filling task, as illustrated in figure 1 @xcite . each tweet text is associated with one of the five event type i listed in casting the event slot filling as a binary classification problem, as suggested in zong et al. ( 2020 ), is illustrated in figure 1 (b), where we show how a separate recognition task is considered for each of the event slots from es i . in this case, for each tweet text span, the recognition task of event slot es i j is identifying whether any of the provided text spans can fill that slot or not. in this way, the event slot filling task is cast as a multi-task binary classification problem. as before, the same text span may fill different event slots, pertaining to different event mentions. because event extraction in this challenge is based on a mapping operation from tweet text spans to event slots, we contemplated methods of representing these spans that could take into account more than deep contextual information. we hypothesized that the representation of the tweet text spans should be specific to each of the event slot recognition tasks depicted in figure 1(b) . while contextual span representations based on the widely used bert model @xcite have been successful in many applications, e.g. end-to-end relation extraction (eberts and ulges, 2020) or coreference resolution @xcite , we believe that representing tweet text spans could be improved when considering modern hopfield networks in which the update rule is the attention mechanism used in the transformer and bert, an idea recently advocated in @xcite . these ideas led the design of our multi-task, event-specific extraction system using bert and hopfield pooling (mt-ese.bhp). ## related work zong et al. (2020) introduced the collection of tweets used for training in this shared-task along with the annotation schema and provided two baselines: a logistic regression model and a multitask bert model. their multi-task bert model utilized the contextual embedding for the @xmath1, while we expand on this span embedding representation with width embeddings and a slot-specific hopfield pooling embedding. we also make use of covid-twitter-bert, as opposed to bert-base used by @xcite . @xcite utilized an unsupervised approach to cluster tweets in which users discuss experiences associated with possible covid-19 symptoms. they used the biterm topic model (btm) @xcite to identify tweet topic clusters, and then manually annotated tweets which fell within relevant topic clusters. in contrast, we made use of the annotated tweets which discuss relevant covid-19 events, as provided by the w-nut 2020 shared task-3, designing a supervised approach capable to extract events and their slots, not only symptom discussions. zhang et al. (2020) manually identified twitter users which appeared depressed and retrieved historical tweets from these users. they attempted to identify whether a twitter user may be depressed based on their public tweet history using the language model xlnet @xcite and they perform user-level classification using a support vector machine (svm). they apply this system to detect and monitor trends of depression during the covid-19 pandemic. we differ from this system by utilizing a domain-specific language model with covid-twitter-bert is trained to extract several types of events and their corresponding slots from tweets related to covid-19. ## results we trained five separate models, for each of the five event types, and they all share the same training schedule and settings: we use an initial learning rate of 2e -5 , warmed up to from 0 over the first epoch and then decayed to 0 linearly for the 9 remaining epochs, with the adam optimizer @xcite . we train with gradient accumulation, where we accumulate the gradients for a batch size of 32 with 4 batches of size 8. we initialize the span width embeddings with a nor- performance on the shared-task was determined based on micro f 1 score for the mapping of text spans into event slots, computed for each event type and overall, on an unseen evaluation set of 2500 tweets with 500 tweets per event type. participating teams were provided the evaluation tweets 5 days in advance of the run submission deadline, and only one submission was allowed per team. our team name and single submission were titled "hltri" and utilized the mt-ese.bhp described in this paper. our system produced an overall f 1 score of 64.76% for mapping event slots across all event types, which accounts for the 2nd-best overall results in the official shared-task evaluation. our system obtained the best results for mapping tweets text spans in the event slots of the tested positive event type, and 3rd-best for the tested negative event type, 2nd-best for the can not test event type, 3rd-best for the death event type, and 4th for cure event type. overall the precision, recall, and f 1 scores for the mapping of text spans into event slots in the official shared-task evaluation are provided in in the table, we also show the best, median, and worst precision, recall, and f 1 scores across all team submissions. we believe that our system performed best on the mapping into event slots for the tested positive event type, because in the training data, we were provided with the most annotated tweets for this event type. we also performed well on mapping into the event slots of the can not test event type, out-performing the median f 1 score by 6.38%. our system was competitive on all but the cure event type, scoring above the median f 1 scores and coming close to the best f 1 scores on all other event types. detailed results are provided in ## discussion the event organizers released the annotated evaluation tweets at the same time they released evaluation results, therefore we were able to analyze event slot-specific performance and perform a qualitative error analysis on specific tweets and event slots which were incorrectly classified. we found that our system performed best on mapping text spans into event slots for the largest event type: tested positive, and within that event type also performed better on average on event slots which had more examples in the training data. while this is partially due to the fact that the tested positive event type has the largest collection of annotated tweets, approximately double that of all other event types, this is also likely due to the fact that most of the training hyper-parameters, listed in section 4, were selected based on initial experimental performance on the training tested positive collection. this hyperparameter selection decision likely biased our overall architecture towards improved performance on mapping of a text span into an event slot for the tested positive event type at the cost of reducing performance in other event types. this hypothesis is supported by our shared-task evaluation ranking, where we came in 1st in tested posi-tive but placed lower in all other event types. event type tweet with text span mt-ese.bhp annotation tested pos. "staffer for [vice president mike pence] × employer tests positive for @xmath2. "reports coming out that the infant child of close contact × [the mother] who died of covid-19 tested negative for covid-19. the sars-cov-2 is mysterious." can not test "[@louisianagov] @ladepthealth still can't × where get tested though." death "@fallofthecabal [i] personally know 3 1st × relation respknders and 2 nurses who have died from complications of covid-19. i have no horse in this race so to speak." cure "at @whitehouse briefing today, a so-called reporter said that who cure × [biden] recommends flying our flag at half-staff. well,, that sure must be the cure for #covid19. my suggestion: wait till after the pandemic is over, then, in honor of those who died, fly the mast at half-staff." cure "@donaldjtrumpjr [you] advocate drinking bleach to cure covid-19. × who cure we are all aware of how much you understand disease." ## conclusion in this paper we described the multi-task eventspecific extraction system using bert and hopfield pooling (mt-ese.bhp) that was developed for the w-nut 2020 shared task 3. our system learned how to take advantage of contextual embeddings such that embeddings for text spans can be learned, while also learning embeddings for each event slot of each event type. this was made possible by using hopfield pooling. the text span embeddings informed binary classifiers (one for each event slot) that decided whether tweet text spans can be mapped into an event slot or not. separate such binary classifiers were trained for each event type. the results that we obtained are promising. these results could be further used to learn how to associate event slots for each mention of an event type in tweets, instead of producing only a bag of filled event slots. this would be a requirement for knowledge extraction from covid-relevant tweets useful for public health applications.