Multi-task Learning to Enable Location Mention Identification in the Early Hours of a Crisis Event (2021)

Abstract: Training a robust and reliable deep learning model requires a large amount of data. In the crisis domain, building deep learning models to identify actionable information from the huge influx of data posted by eyewitnesses of crisis events on social media, in a time-critical manner, is central for fast response and relief operations. However, building a large, annotated dataset to train deep learning models is not always feasible in a crisis situation. In this paper, we investigate a multi-task learning approach to concurrently leverage available annotated data for several related tasks from the crisis domain to improve the performance on a main task with limited annotated data. Specifically, we focus on using multi-task learning to improve the performance on the task of identifying location mentions in crisis tweets.

## introduction social media has evolved into a platform for people to share their concerns, report information as eyewitnesses of events, and also call for help, especially during crisis situations. the huge amount of data that is posted on social media during crisis events could be used to build reliable and robust deep learning models for identifying information useful to crisis management and response teams. however, using social media data for a particular task, oftentimes, requires intensive manual effort in the form of annotation. the effort becomes even more arduous when we consider the noisy nature of social media content and the amount of labelled data required for a typical deep learning model. the domain of crisis-related social media analysis, tweets in particular, is a well-researched field with labelled data available for various tasks @xcite @xcite . however, most of the available humanannotated datasets consists of thousands of instances, at best, which means that crisis datasets are relatively small compared to those available for tasks in other domains. furthermore, for tasks that can support a new, emergent crisis situation, human-labelled data of large volume cannot be acquired for the reasons discussed above. in this work, we explore ways in which we can harness the available small datasets from the domain to bolster performance for individual tasks of interest. one popular approach in addressing the sizelimitation of labelled data for a particular task is to leverage unlabelled data. in the field of natural language processing (nlp), the recent advancements in transformer-based architectures, and associated pre-training with huge amounts of unlabelled data, has largely been successful in addressing this issue. transformer-based architectures, currently, hold state-of-the-art results for many nlp tasks. however, the domain shift from the pre-training corpus to a downstream task's domain is still a significant issue @xcite . moreover, further pre-training with domain-specific unlabelled data is compute-intensive @xcite and it is not always feasible. an alternative approach to address the limitation in terms of labeled data is to concurrently leverage smaller datasets available for different, but related tasks using a multi-task learning strategy @xcite . in the multi-task setting, some layers can be shared across different tasks, while each task can also have one or more task-specific layers, and the entire model is trained in parallel for all the tasks. a multi-task model is designed with the intuition that the lower layers of the model learn abstract features common to related tasks, while the upper layers learn features specific to each individual task. this approach is especially useful in the domain of crisis-related social media analysis, given the lack of large datasets, while smaller datasets are available for different tasks. in this work, our main focus is on the task of identifying fine-grained locations from tweet texts. identification of location entities in tweets posted during crisis events is vital in extracting actionable situational awareness information. furthermore, identifying the entities according to a hierarchy of location types can help in geographical location disambiguation and geo-coding. we use the english subset of the dataset published by @xcite , which has only a few thousands instances. to address the issue of limited data size, we use a multi-task learning setting to augment the learning of fine-grained location identification with three other tasks in the domain of crisis-related tweets: key-phrase identification (chowdhury et al., 2020), eyewitness-account classification @xcite and humanitarian categories classification @xcite . we hypothesize that the similar nature of the tasks and the common abstract objective of identifying actionable information in crisis-related tweets will result in a performance boost for the main task considered, i.e., identification of fine-grain locations in tweet texts. ## related work multi-task learning is a well-researched topic in the field of nlp, and deep learning, in general @xcite . @xcite outlines one of the popular strategies for implementing multi-task learning: hard parameter sharing. in hard parameter sharing, a module shared by all tasks is followed by task-specific modules. on the other hand, in soft parameter sharing @xcite , each task has its own set of layers, but the parameters of the task-specific layers are constrained to be similar across tasks to enforce exchange of information among tasks. hard parameter sharing is known to reduce over-fitting @xcite , and is thus useful for tasks with small train-ing sets. the usefulness of multi-task learning has been shown in a variety of applications, including image @xcite , voice @xcite ) and text @xcite @xcite analysis. in the crisis domain, @xcite presented a multi-modal multi-task model using a single multi-modal dataset containing labels for different tasks. chowdhury et al. (2020) used single-token keywords identification as an auxiliary task when predicting multi-token keyphrases. the work by @xcite is the closest to our work, in that they used separate taskspecific datasets in a multi-task learning setting. however, they performed experiments on general nlp benchmark datasets, such as glue @xcite . ## background and approaches we use the hard-parameter sharing approach for multi-task learning, where task-specific modules (τ ) are attached on top of a shared module (γ) as shown in figure 1 . we use the algorithm proposed by @xcite to train multiple task modules, in parallel. ## results and discussion as can be seen in transformer average performance improvement (%) @xmath0.78 3.43 roberta 4.75 2.36 albert 4.17 6.35 ## conclusions and future work in this paper, we studied the effect of using other closely related tasks in a multi-task setting for the task of fine-grained location identification in the domain of crisis-related tweets. our results show that using multi-task learning can improve the performance significantly as compared to single-task learning. the approach could be specially useful when analysing an emergent crisis, where annotating large amount of data is not feasible and the entire analysis process is time-critical. moreover, our results also show that when a small amount of data is available, the model may benefit more from using other related tasks rather than using additional data for the same type of task (i.e., two location identification tasks). in addition to studying the effect of multi-task learning, we also studied the effect of different transformer variants on the performance of the model for the main task. with different variants available, it is necessary to find the best-fitting one for the task at hand. the results show that the choice of the shared module has significant effect on the performance. as part of future work, we plan to explore other multi-tasking and domain adaptation strategies with different collections of datasets to allow for stress testing of the approach in terms of generalization. given the promising results with multi-task learning, we plan to perform studies with respect to the nature of the crisis for the dataset. this will allow us to focus on specific datasets for specific crises and potentially improve the state-of-the-art on those datasets.