D e C risis MB : Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank (2023)

Abstract: During crisis events, people often use social media platforms such as Twitter to disseminate information about the situation, warnings, advice, and support. Emergency relief organizations leverage such information to acquire timely crisis circumstances and expedite rescue operations. While existing works utilize such information to build models for crisis event analysis, fully-supervised approaches require annotating vast amounts of data and are impractical due to limited response time. On the other hand, semi-supervised models can be biased, performing moderately well for certain classes while performing extremely poorly for others, resulting in substantially negative effects on disaster monitoring and rescue. In this paper, we first study two recent debiasing methods on semi-supervised crisis tweet classification. Then we propose a simple but effective debiasing method, DeCrisisMB, that utilizes a Memory Bank to store and perform equal sampling for generated pseudo-labels from each class at each training iteration. Extensive experiments are conducted to compare different debiasing methods’ performance and generalization ability in both in-distribution and out-of-distribution settings. The results demonstrate the superior performance of our proposed method. Our code is available at https://github.com/HenryPengZou/DeCrisisMB.

## introduction during natural disasters, real-time sharing of crisis situations, warnings, advice and support on social media platforms is critical in aiding response organizations and volunteers to enhance their situational awareness and rescue operations @xcite . although existing works utilize such information to build models for crisis event analysis, standard supervised approaches require annotating vast amounts of data during disasters, which is impractical due to limited response time @xcite @xcite @xcite @xcite . on the other hand, current semi-supervised models can be biased, performing moderately well for certain classes while extremely worse for others, resulting in a detrimentally negative effect on disaster monitoring and analysis @xcite @xcite @xcite . for instance, neglecting life-essential classes, such as requests or urgent needs, displaced people & evacuations and injured or dead people, can have severely adverse consequences for relief efforts. therefore, it is crucial to mitigate bias in semi-supervised approaches for crisis event analysis. in this paper, we investigate and observe that bias in semi-supervised learning can be related to interclass imbalances in terms of numbers and accuracies of pseudo-labels produced during training. we do this analysis using a representative work in semisupervised learning, pseudo-labeling (psl) @xcite @xcite . we then study two different debiasing methods for semi-supervised learning on the task of crisis tweet classification. these two state-of-the-art semisupervised debiasing approaches are: debiasing via logits adjustment (logitadjust) @xcite and debiasing via self-adaptive thresholding (sat) @xcite . our analysis show that although these methods have effects in debiasing and balancing pseudo labels across classes, their debiasing performance is still unsatisfying and there are drawbacks that need to be addressed. logitadjust debiases the pseudo-labeling process by explicitly adjusting predicted logits based on the average probability distribution over all unlabeled data. however, we observe that this explicit adjustment makes it difficult for models to fit data, leading to unstable training, and their classwise pseudo-labels are still highly imbalanced. sat proposes to dynamically adjust global and local thresholds of pseudo-labeling to enforce poorlylearned categories generating more pseudo-labels. this approach does help produce more balanced pseudo-labels but comes at the cost of sacrificing the accuracy of the pseudo-labels (in-depth analysis and visualized comparisons are provided in section 6.) to address these issues, we propose a simple but effective debiasing method, decrisismb, that utilizes a memory bank to store generated pseudolabels. we then use this memory bank to sample an equal number of pseudo-labels from each class per training iteration for debiasing semi-supervised learning. extensive experiments are conducted to compare the three debiasing methods for semisupervised crisis tweet classification. additionally, we evaluate and compare their generalization ability in out-of-distribution datasets and visualize their performance in debiasing semi-supervised models. our results and analyses demonstrate the substantially superior performance of our debiasing methods. the contributions of this work are summarized as follows: ## analysis of inter-class bias even with balanced datasets, semi-supervised learning may exhibit bias towards certain classes while ignoring others. in this section, we demonstrate that such bias can be related to inter-class imbalances, including the numbers and accuracies of pseudo-labels generated during training. this motivates us to debias semi-supervised learning by balancing pseudo-labels during training. ## debiasing methods in this section, we first introduce our baseline pseudo-labeling (psl) @xcite @xcite , a representative semisupervised method. we then present and discuss two state-of-the-art semi-supervised debiasing approaches: debiasing via logits adjustment (log-itadjust) in @xcite and debiasing via self-adaptive thresholding (sat) in @xcite . finally, we introduce our debiasing method decrisismb. ## analysis to further understand each debiasing approach and why the proposed method achieves the best debasing results, we provide several visualizations on hurricane datasets. figure 3 demonstrates qualitative comparisons of different debiasing methods. figure 4 shows the comparison on total numbers of produced pseudo-labels. figure 5 presents the average pseudo-label accuracy over the worst 1, 4 and 8 classes. overall, all three debiasing methods have some effects in debiasing and balancing pseudo-label numbers across the class, as shown in figure 3 , 4. however, there are some differences: ## conclusion in this work, we demonstrate that semi-supervised models and their pseudo-labels generated on social media data posted during crisis events can be biased, and balancing pseudo-labels used in training can effectively debias semi-supervised models. we then study and compare two recent debiasing approaches in semi-supervised learning with our proposed debiasing method for crisis tweet classification. experimental results show that our method based on memory bank and equal sampling achieves the best debiasing results quantitatively on both in-distribution and out-of-distribution settings. we believe our work can serve as a universal and effective adds-on debiasing module for semisupervised learning in different domains.