C omplex D ata L ab at W - NUT 2020 Task 2: Detecting Informative COVID -19 Tweets by Attending over Linked Documents (2020)

Abstract: Given the global scale of COVID-19 and the flood of social media content related to it, how can we find informative discussions? We present Gapformer, which effectively classifies content as informative or not. It reformulates the problem as graph classification, drawing on not only the tweet but connected webpages and entities. We leverage a pre-trained language model as well as the connections between nodes to learn a pooled representation for each document network. We show it outperforms several competitive baselines and present ablation studies supporting the benefit of the linked information. Code is available on Github.

## introduction covid-19 is a critical public health crisis, with over 25 million cases and 840 thousand deaths worldwide and counting @xcite . one important way to fight the pandemic is through understanding and leveraging the vast amount of information humans produce related to it on social media. success here can facilitate individual and overall case detections, contact tracing, case predictions, effective information dissemination, and more @xcite . however, the data is far too big -for example, hundreds of millions of tweets @xcite -to extract all the useful information by hand. there is also a huge amount of useless and even outright false information @xcite . therefore, algorithms that extract useful information effectively are crucial. wnut-2020 task 2 @xcite is to build such a system on a particular twitter dataset, and predict which tweets are informative "about 1 https://github.com/complexdata-mila/ gapformer recovered, suspected, confirmed and death cases as well as location or travel history of the cases." we present here gapformer, which combines multiple sources of information -the tweet, content of linked web pages, and information about named entities related to the tweet -to improve detection of informative content. our experiments show gapformer outperforms 7 baselines. it also performs best when incorporating all three types of data, showing the efficacy of using the proposed graph structure. we believe this method can also be applied in other related settings, such as misinformation detection, and have work in progress to investigate possible extensions. ## background and related work before covid-19, mining information from tweets about disaster events has been considered in several contexts. the crisisnlp @xcite collection has twitter datasets from 19 crises, including diseases ebola and mers. there are also works analyzing particular events @xcite @xcite and methods @xcite @xcite . however, covid-19 is unprecedented in its global scale and impact. while there is also a historic amount of research taking place to counter it @xcite , more is needed -there are still lives to be saved. in addition, while people have tackled related covid-19 twitter data tasks @xcite @xcite , to our knowledge this particular task formulation has not been thoroughly investigated prior to this shared task competition. with the success of transformer-based models @xcite , it is natural to consider applying them here. bertweet (dat quoc nguyen and nguyen, 2020) and covidtwitter-bert @xcite , bert language models fine-tuned on twitter data (and the latter on tweets related to covid-19 specifically), are particularly relevant. however, pure language models struggle to consider additional contextual information beyond the raw content of the tweet. as our experiments show, this is an important limitation. because this limitation applies to many other applications as well, a rapidly growing area of research is embedding additional contextual knowledge into language models, especially with the use of graph structures @xcite @xcite . this is often done at the language modelling level, such as how @xcite uses normalized mutual information to inject global information into each layer of bert. that said, other work has focused on injecting knowledge into an existing pre-trained language model. in particular, transformer-xh @xcite proposes a way to leverage the network structure of documents alongside a pre-trained model for a classification task. gapformer builds on this line of work by proposing a simple yet effective architecture which contextualizes each node before pooling the graph into a single fixed embedding. ## conclusions we presented gapformer, which detects informative covid-19 tweets, using a graph classification system that leverages articles and entities. this is done by first building graphs using normalized pointwise mutual information to determine related nodes, and then combining pre-trained language models with graphsage convolutions and a pooling mechanism to classify them. as shown in the experiments, this way of incorporating additional data is effective. it enables this system to outperform all seven baselines tested. in future work, we plan to extend and upgrade this system in several ways: we are also looking at real-world applications of this tool. since accuracy and other metrics are already above 90%, we believe it can have practical value, especially with the improvements above. we aim therefore to find direct ways to use gap-former to help mitigate the covid-19 pandemic.