Multilingual Detection of Personal Employment Status on T witter (2022)

Abstract: Detecting disclosures of individuals’ employment status on social media can provide valuable information to match job seekers with suitable vacancies, offer social protection, or measure labor market flows. However, identifying such personal disclosures is a challenging task due to their rarity in a sea of social media content and the variety of linguistic forms used to describe them. Here, we examine three Active Learning (AL) strategies in real-world settings of extreme class imbalance, and identify five types of disclosures about individuals’ employment status (e.g. job loss) in three languages using BERT-based classification models. Our findings show that, even under extreme imbalance settings, a small number of AL iterations is sufficient to obtain large and significant gains in precision, recall, and diversity of results compared to a supervised baseline with the same number of labels. We also find that no AL strategy consistently outperforms the rest. Qualitative analysis suggests that AL helps focus the attention mechanism of BERT on core terms and adjust the boundaries of semantic expansion, highlighting the importance of interpretable models to provide greater control and visibility into this dynamic learning process.

## introduction up-to-date information on individuals' employment status is of tremendous value for a wide range of economic decisions, from firms filling job vacancies to governments designing social protection systems. at the aggregate level, estimates of labor market conditions are traditionally based on nationally representative surveys that are costly to produce, especially in low-and middle-income countries @xcite . as social media becomes more ubiquitous all over the world, more individuals can now share their employment status with peers and unlock the social capital of their networks. this, in turn, can provide a new lens to examine the labor market and devise policy, especially in countries where traditional measures are lagging or unreliable. a key challenge in using social media to identify personal disclosures of employment status is that such statements are extremely rare in an abundance of social media content -roughly one in every 10,000 posts -which renders random sampling ineffective and prohibitively costly for the development of a large labeled dataset. on the other hand, simple keyword-based approaches run the risk of providing seemingly high-accuracy classifiers while substantially missing linguistic variety used to describe events such as losing a job, looking for a job, or starting a new position (see figure 1 for example). in the absence of a high-quality, comprehensive, and diverse ground-truth about personal employment disclosures, it is difficult to develop classification models that accurately capture the flows in and out of the labor market in any country, let alone robustly estimating it across multiple countries. furthermore, state-of-the-art deep neural models provide little visibility into or control over the linguistic patterns captured by the model, which hampers the ability of researchers and practitioners to determine whether the model has truly learned new linguistic forms and sufficiently converged. active learning (al) is designed for settings where there is an abundance of unlabeled examples and limited labeling resources @xcite . it aims to focus the learning process on the most informative samples and maximize model perfor-mance for a given labeling budget. in recent years, al proved successful in several settings, including policy-relevant tasks involving social media data @xcite . the success of pre-trained language models such as bert @xcite in a variety of language understanding tasks has sparked interest in using al with these models for imbalanced text classification. yet, most research in this field has focused on artificially-generated rarity in data or imbalance that is not as extreme as the present setting @xcite . therefore, there is no evidence of the efficiency of al using bert-based models for sequence classification in real-world settings with extreme imbalance. it is unclear whether some al strategies will perform significantly better than others in these settings, how quickly the different strategies will reach convergence (if at all), and how the different strategies will explore the linguistic space. in this work, we leverage bert-based models @xcite in three different al paradigms to identify tweets that disclose an individual's employment status or change thereof. we train classifiers in english, spanish, and portuguese to determine whether the author of a tweet recently lost her job, was recently hired, is currently unemployed, posting to find a job, or posting a job offer. we use two standard al strategies, uncertainty sampling @xcite and adaptive retrieval @xcite , and propose a novel strategy we name exploit-explore retrieval that uses k-skip-n-grams (n-grams with k skipped tokens) to explore the space and provide improved interpretability. we evaluate the models both quantitatively and qualitatively across languages and al strategies, and compare them to a supervised learning baseline with the same number of labels. therefore, our contributions are: ## discussion and conclusion this work developed and evaluated bert-based models in three languages and used three different al strategies to identify tweets related to an individual's employment status. our results show that al achieves large and significant improvements in precision, expansion, and diversity over stratified sampling with only a few iterations and across languages. in most cases, al brings an order of magnitude more positives while preserving or improving both the precision and diversity of results. despite using fundamentally different al strategies, we observe that no strategy consistently outperforms the rest. within the extreme imbalance setting, this is in line with -and complements -the findings of @xcite . additionally, our qualitative analysis and exploration of the exploit-explore retrieval give further insights into the performance improvements provided by al, finding that substantial amounts of queried tweets hone the model's focus on employment rather than surrounding context and expand the variety of motifs identified as positive. this puts exploit-explore retrieval as a valuable tool for researchers to obtain greater visibility into the al process in extreme imbalance cases without compromising on performance. while the present work demonstrates the potential of al for bert-based models under extreme imbalance, an important direction for future work would be to further optimize the al process. one could for instance study the impact on performance of the stratified sample size or the al batch size. to overcome the poor seed quality for some classes, other seed generation approaches could be tested, such as mining online unemployment forums using topic modeling techniques to discover different ways to talk about unemployment. in terms of model training and inference, the use of multitask learning for further performance improvement could be studied due to the fact that classes of unemployment are not mutually exclusive. we hope that our experimental results as well as the resources we make available will help bridge these gaps in the literature.