Multi Output Learning using Task Wise Attention for Predicting Binary Properties of Tweets : Shared-Task-On-Fighting the COVID -19 Infodemic (2021)

Abstract: In this paper, we describe our system for the shared task on Fighting the COVID-19 Infodemic in the English Language. Our proposed architecture consists of a multi-output classification model for the seven tasks, with a task-wise multi-head attention layer for inter-task information aggregation. This was built on top of the Bidirectional Encoder Representations obtained from the RoBERTa Transformer. We were able to achieve a mean F1 score of 0.891 on the test data, leading us to the second position on the test-set leaderboard.

## introduction in recent years, the spread of misinformation on social media has been growing rapidly. amid the global pandemic, the spread of misinformation has had serious consequences. covid-19 misinformation caused mass hysteria and panic; reluctance to use masks and follow social distancing norms; denial of the existence of @xcite there is a need for automatic detection and flagging of tweets spreading misinformation. automatic detection of misinformation is an open research problem in nlp. misinformation is intentionally written to deceive and pass as factual which makes detecting it a difficult task. @xcite analyzed a dataset of 505k tweets. they used sentiment analysis, polarity scores, and liwc to build features. they used ml models such as randomforest, adaboost, svm, etc to classify tweets as factual/misinformation. predicting answers to multiple questions, which is the setup of our current problem statement, can be modeled as a multi-task-learning problem. (crawshaw, 2020) have highlighted different methods for sharing information among tasks, for task-specific performance boosts, such as cross-stitching and @xcite have highlighted the use of attention mechanisms for the multi-task learning problem and show that it performs competitively with other approaches. inspired by this idea, we propose an attention-based architecture roberta multihead attn for our task by incorporating inter-task information for task-specific performance enhancement. with a test-set f1 score of 0.891, our approach shows the superiority of combining information among tasks over modeling them independently and shows the effectiveness of multihead-attention for this purpose. ## task description and dataset the objective of this task @xcite is to predict various binary properties of tweets. we were given several tweets where we had to predict answers to 7 questions. these questions pertain to whether the tweet is harmful, whether it contains a verifiable claim, whether it may be of interest to the general public, whether it appears to contain false information, etc. there were three tracks of languages on english, arabic, and bulgarian and a team was free to choose any subset of the languages. for the english language, the training dataset consisted of 862 tweets 1 the dev set consisted of 53 tweets and the testing set consisted of 418. the training dataset statistics are shown in ## methodology we explored different base embeddings inspired by @xcite and @xcite . these are state-of-the-art language models which when used with task-specific fine-tuning, perform well on a wide variety of tasks. the embeddings are passed to our task-specific architecture for further processing and the whole model is trained end-toend. we hypothesize that the prediction for one question may benefit from the use of information needed to predict another question. for instance, questions q4 ("harmfulness: to what extent is the tweet harmful to the society/person/company/product?") and q6 ("harmful to society: is the tweet harmful to the society and why?") in the task have a deduction process that may have several common steps. to model this, we add an inter-task multi-head attention layer before our prediction heads. ## conclusion and future work in this paper, we have described our system for predicting different binary properties of a tweet, on the english sub-task of the shared task on fighting the covid infodemic in nlp4if'21. our approach uses the roberta base architecture for the initial embeddings and builds on top of that using taskwise multi-head attention layers. our results show that using a multi-head attention approach for aggregating information from different tasks leads to an overall improvement in performance. possible developments in this task can include the incorporation of additional contextual information in our models using tweet-related features like the #(number) of urls in a tweet, the # of user mentions and the # of hashtags, etc. also, user-related features such as the # of followers, account age, account type (verified or not) can be included. these features contain a lot of auxiliary information that can aid in fine-grained classification. sociolinguistic analysis such as linguistic inquiry and word count (liwc) can also be used to gather emotional and cognitive components from the tweet.