UCD - CS at W - NUT 2020 Shared Task-3: A Text to Text Approach for COVID -19 Event Extraction on Social Media (2020)

Abstract: In this paper, we describe our approach in the shared task: COVID-19 event extraction from Twitter. The objective of this task is to extract answers from COVID-related tweets to a set of predefined slot-filling questions. Our approach treats the event extraction task as a question answering task by leveraging the transformer-based T5 text-to-text model. According to the official evaluation scores returned, namely F1, our submitted run achieves competitive performance compared to other participating runs (Top 3). However, we argue that this evaluation may underestimate the actual performance of runs based on text-generation. Although some such runs may answer the slot questions well, they may not be an exact string match for the gold standard answers. To measure the extent of this underestimation, we adopt a simple exact-answer transformation method aiming at converting the well-answered predictions to exactly-matched predictions. The results show that after this transformation our run overall reaches the same level of performance as the best participating run and state-of-the-art F1 scores in three of five COVID-related events. Our code is publicly available to aid reproducibility

## introduction since the outbreak of covid-19, a wide variety of research has been conducted to mine insights or gain rapid access to information in relation to the crisis. for example, the trec-covid challenge seeks advanced techniques for finding useful information from the cord-19 corpus comprising hundreds of thousands of covid-related academic articles @xcite . related research focuses on mining valuable information from social media @xcite . this is largely inspired by the fact that social media motivates people to share information or express their opinions quickly. however, this user-generated content is usually noisy and enormous during crises, and thus needs to be condensed and filtered before further processing and analysis. motivated by this, the acl workshop on noisy user-generated text 2020 proposed a shared task (w-nut task-3) of extracting covid-19 related events from twitter @xcite . the major objective of this task is to seek computational linguistic techniques for extracting text spans from a corpus of raw tweets to answer a set of predefined slot questions. the corpus used in the task can be described in two parts. first, the corpus consists of approximately 7,500 tweets categorised into five broad event types: (1) tested positive, (2) tested negative, (3) can not test, (4) death and (5) cure and prevention. as the names indicate, these tweets are a summary of people's primary concerns about covid-19 that they likely post about on social media. secondly, for the tweets in each event, a set of questions or slot-filling types are defined to help gather more fine-grained information about the tweets. the human annotations, (i.e., ground truths) of the corpus are simply the answers to the predefined questions. figure 1 illustrates an example from this corpus. this example represents a tested pos-itive or tested negative event, and the associated slot-filling questions relate to the "who" and the "duration" slots, asking who has tested positive or negative and how long it takes to know the test result. the annotation process in part 2 is conducted by annotators who select answers from a drop-down list of candidate choices foot_1 . this ex- ## related work the literature has seen much work on processing crisis-related messages on social media. for example, the trec incident streams @xcite track is a research initiative for seeking computational linguistic techniques for finding actionable information from twitter during crises. with similar motivations to this initiative, many techniques have been applied for crisis messages categorisation and analysis in recent years. for instance, @xcite apply label embedding for crisis tweet classification adopting a bi-directional lstm model. wang and lillis (2020) leverage contextual elmo embeddings @xcite and data augmentation. crisisbert is proposed for crisis event detection @xcite through fine-tuning the pre-trained bert @xcite . more recently, the covid-19 pandemic has motivated research on insights mining, information extraction and classification on social media. since the pandemic is characterised by being longlived, unlike more short-term crises such as earthquakes or shootings, the messages on social media are studied to gain a better understanding of the pandemic from multiple perspectives. for example, tweetscov19 @xcite ) is a corpus of semantically annotated tweets about covid-19 that can potentially be used to explore public opinion and perception on the pandemic. @xcite introduce covid-twitter-bert, which pre-trains a bert model on hundreds of millions of covid-related tweets and then fine-tunes the model to downstream tasks including vaccine sentiment analysis and vaccine stance classification. although most recent work involves finetuning transformer-based models for covidrelated short message processing, we have chosen to take an alternative approach for this task. inspired by the text-to-text t5 model, our approach formulates the w-nut task 3 as a questionanswering problem. the principal idea is that its unified source and target structure can be easily adapted to other domain-similar tasks such as informativeness classification or vaccine sentiment for covid-related tweets. regarding fine-tuning t5 for covid-related language tasks, probably the most relevant work is that @xcite fine-tuned t5 for covid-related question answering from scientific articles. however, they con- transformer encoder transformer decoder source sequence: sequence construction: concatenate & uncase datasets *prince charles tests positive for corona prince william knowing he's the next in line to the throne: https://t.co/b1nmiplj69 . who is tested positive? author of the tweet, not specified, the next in line, line to the throne, *prince charles, corona prince william, he, the next, line, the throne. candidate choices slot question tweet text context: question: choices: </@xmath0: <@xmath1: available when training structed the source sequence by concatenating a document and a query, and the target sequence is a generated token indicating the document's relevance to the query. our approach is characterised by being capable of unifying various text classification tasks to a general question answering task. in our approach, the source sequence takes the raw text as the context, a classification description as the question and the classification's labels as the candidate choices. the target sequence is simply the generated answers that are exact text spans from the source sequence, indicating which label(s) the text belongs to. ## method in this section, we detail our approach in w-nut task-3. we firstly describe how a general transformer encoder-decoder model work in a sequence-to-sequence way and then introduce our approach adapting it to the event extraction task. basically, given a source sequence x : {x 1 , x 2 , ..., x n }, the transformer encoder uses it as the input to output its contextualised encoding sequence x : {x 1 , x 2 , ..., x n }. the encoding process can be represented by a mapping function f with learnable parameters θ e , as follows. the encoded source sequence then is used as the input to the transformer decoder that defines the conditional probability distribution of a target sequence y : {y 1 , y 2 , ..., y m } given x 1:n : where p with learnable parameters θ d is the function to learn the conditional probability distribution by the decoder. the target token y i is generated conditional on both the source context x 1:n and its previous i -1 already-generated tokens in an auto-regressive way. the powerful part of the transformer-based decoder is that it applies a single directional attention mechanism, i.e., masked multi-head attention as proposed by @xcite to learn the relations between the target token and its previous tokens. meanwhile, it applies a bi-directional attention mechanism, i.e., multihead attention, to learn the relations between the target token and the source context, which is also used in the encoder to learn the relations (selfattentions) between the source tokens. inspired by the feature of target generation given a source in a transformer encoder-decoder model, we leverage it for the shared task. figure 2 presents the system architecture of our approach. the core component of the system is the sequence construction, which converts each tweet in the dataset into a source sequence and target sequence that well fits to train the transformer encoder-decoder model in a text-to-text fashion. as introduced, each tweet in the dataset is annotated in two parts, 1), labels indicating the event types, 2), answers to the slot questions. our approach uses both parts of the annotations to construct the source and target sequences as follows: although the aforementioned only details the 3 the event type questions are manually constructed: taking death as an example: does this tweet report death from coronavirus? after the source and target sequences are constructed, the next step is to train a transformer encoder-decoder model. as evidenced in the literature, sequence-to-sequence transformers such as bart @xcite or t5 @xcite has demonstrated great success in various downstream language tasks via transfer learning. since t5 has different pre-trained weights available where in particular its small version is easy to handle in the pilot study, we set up the encoderdecoder model following the t5 @xcite architecture and train it on this task via finetuning its pre-trained weights. ## experiments this section describes the details of experimental process and results. ## conclusion this paper presents our text-to-text based approach at w-nut 2020 shared task 3. we show that the principal idea behind the approach is adaptability to other domain-similar tasks such as informativeness classification of covid-19 tweets. we expect to conduct more work on this adaptability in the future. it is even more interesting to test the idea in zero-shot learning. for example, how well it performs if transferring the model that is trained on the event extraction corpus to do inference in the informativeness task directly without further training. in addition, we empirically present that our system is effective, achieving competitive performance and arguably the stateof-the-art f1 scores in three of five covid-events in the shared task. despite the effectiveness, one concern of our approach is the model size. our best performed model is fine-tuned using the large version of t5 with around 770m parameters @xcite . this makes it important to compress the model efficiently in the future.