NHK _ STRL at WNUT -2020 Task 2: GAT s with Syntactic Dependencies as Edges and CTC -based Loss for Text Classification (2020)

Abstract: The outbreak of COVID-19 has greatly impacted our daily lives. In these circumstances, it is important to grasp the latest information to avoid causing too much fear and panic. To help grasp new information, extracting information from social networking sites is one of the effective ways. In this paper, we describe a method to identify whether a tweet related to COVID-19 is informative or not, which can help to grasp new information. The key features of our method are its use of graph attention networks to encode syntactic dependencies and word positions in the sentence, and a loss function based on connectionist temporal classification (CTC) that can learn a label for each token without reference data for each token. Experimental results show that the proposed method achieved an F1 score of 0.9175, out- performing baseline methods.

## introduction the outbreak of covid-19 that has occurred since the end of 2019 has greatly impacted our daily lives. in these circumstances, it is important for everyone to understand the situation and grasp the latest information to avoid causing too much fear and panic. nowadays, social networking sites (snss) such as twitter and facebook are important information sources because users post information regarding their personal events-including that related to covid-19-in real time. for this reason, many monitoring systems for covid-19 have been developed such as the johns hopkins coronavirus dashboard foot_0 and the covid-19 health system response monitor foot_1 . many systems use snss as resources, but largely depend on manual work such as using cloud sourcing to extract informative posts from massive numbers of uninformative ones. generally, snss contain too much information on miscellaneous topics, so extracting important information is difficult. therefore, we attempted to develop a method to extract important information. our method first embeds each token in the input sentence using bert @xcite . then, the vectors are fed into graph attention networks (gats) @xcite to encode tokento-token relations. finally, our method classifies each vector into labels using feed-forward neural networks (ffnns). in the training process, we use a loss function based on connectionist temporal classification (ctc) @xcite . experimental results show that our method using gats and the ctc-based loss function achieved an f1 score of 0.9175, outperforming baseline methods. our contributions are as follows: (1) we propose a gat-based network to embed syntactic dependencies and positional features of tokens in an input sentence. (2) we also propose a loss function, which enables to train labels for each token. (3) we confirmed the effectiveness of our proposed methods using the identification of informative covid-19 english tweets shared task dataset. ## identifying informative covid-19 the identification of informative covid-19 english tweets foot_2 is a shared task held at w-nut (workshop on noisy user-generated text) 2020 @xcite . the purpose of the task is to identify whether english tweets related to covid-19 are informative or not. the dataset for the task contains 7,000 tweets for training, 1,000 for validating, and 2,000 for testing. each tweet in the data, excluding those in the testing data are labelled informative or uninformative. the target metric of the task is the f1 score for informative tweets. token embedding 1200 new cases in the uk syntactic dependency embedding syntactic dependency and positional features gat -2 -1 0 1 2 3 ℎ 0 (0) ℎ 1 (0) ℎ 2 (0) ℎ 3 (0) ℎ 4 (0) ℎ 5 (0) * example distance from ℎ 2 (0) position encodings ℎ 𝑛 (0) ℎ 𝑛 (𝐿) labels for each token classification *one or more "informative" labels included, then output is "informative." ## methods the overview of our method is illustrated in figure 1 . the key features of our method are embedding syntactic dependencies and positional features using gats @xcite , and calculating loss in the training process using a loss function based on ctc @xcite . we use masked-token estimation as multi-task learning to help improve the generalization capability. we use word-dropout @xcite before bert, and the "dropped" tokens are used as masked words to be estimated in the training process as a multi-task. ## related work there are a number of methods that use gats with a pre-trained language model. lu et al. (2020) use 5 https://competitions.codalab.org/ competitions/25845#results a network on a vocabulary graph, which is based on word co-occurrence information, and @xcite and @xcite use syntactic features as a graph. also, there are several methods that use positional encoding into gats @xcite . our method uses gats to consider syntactic features with positional features in combination, which is distinguishable from conventional methods. the ctc loss function is widely used for long data sequence with not-one-to-one-aligned reference data such as speech recognition @xcite , but to the best of our knowledge, no method that uses ctc for text classification tasks exists. ## conclusion and future work in this paper, we proposed a gats-based model that embeds token-to-token relations, and a loss function that can learn classes for each tokens. we conducted evaluations using the identification of informative covid-19 english tweets dataset, and confirmed that our proposed methods are effective. to determine whether ctctc can work for other tasks especially for the classification into large amount of classes and to exploit pre-trained models other than bert, especially for tweetspecific models such as bertweet @xcite and ct-bert @xcite , are subjects of as our future work.