SMM 4 H ’24 Task6 : Extracting Self-Reported Age with LLM and BERT weet: Fine-Grained Approaches for Social Media Text (2024)

Abstract: The paper presents two distinct approaches to Task 6 of the SMM4H’24 workshop: extracting self-reported exact age information from social media posts across platforms. This research task focuses on developing methods for automatically extracting self-reported ages from posts on two prominent social media platforms: Twitter (now X) and Reddit. The work leverages two ways, one Mistral-7B-Instruct-v0.2 Large Language Model (LLM) and another pre-trained language model BERTweet, to achieve robust and generalizable age classification, surpassing limitations of existing methods that rely on predefined age groups. The proposed models aim to advance the automatic extraction of self-reported exact ages from social media posts, enabling more nuanced analyses and insights into user demographics across different platforms.

## introduction the widespread use of social media platforms by individuals across demographics offers a unique opportunity to gain valuable insights into their health experiences and perspectives. effectively leveraging social media data for research purposes necessitates the development of methods for automatically extracting demographic information, such as user age, with high accuracy. existing methods for identifying user age on social media platforms often rely on categorizing users into predefined age groups. as computational analysis offers new possibilities for investigating complex subjects through social media data, models are being created to automatically identify demographic information, such as the age of users @xcite . many studies have tackled age detection through automated methods, primarily using binary or multi-class classification of predetermined age categories @xcite . these approaches typically involve identifying users' ages from their posts, profile information, or external data sources and subsequently predicting their age groups based on various factors such as their social media activity profile details or a combination of both @xcite ; however, the diversity and inconsistency in the number and scope of age groups used across studies indicate that these methods may not be universally applicable to all scenarios. accurately pinpointing the exact age of social media users, instead of placing them into broad age categories, would enable the extensive use of social media data for applications needing precise age information. this would be particularly beneficial for identifying specific age-related risk factors in observational studies, which current binary or multi-class models fail to address. this work applies two strategies, one using mistral-7b-instruct-v0.2 large language model (llm) and another pre-trained language model bertweet for automatically extracting self-reported exact age information from social media posts on two prominent platforms: twitter and reddit. the proposed models aim to address the issues of existing methods by directly identifying the user's exact age as expressed in the text, enabling the largescale utilization of social media data for a wider range of research applications. \documentclass declaration and before \begin{document}) using \usepackage{graphicx}. ## data description the training dataset consists of 8,800 labeled tweets and 100,000 unlabeled reddit posts, primarily focused on age-related information. the labeled tweets indicate whether the user's exact age could be determined from the text, with "1" indicating explicit or inferred age and "0" otherwise. validation set includes 2,200 labeled tweets and 1,000 reddit posts about dry eye disease, while the testing set includes 2,200 labeled tweets, 2,000 1 106 reddit dry eye disease posts, and 12,482 reddit posts about social anxiety among individuals aged 13-25. ## methodology using bertweet for the second attempt, a bertweetbase (nguyen et al., 2020) implemented using the huggingface toolkit @xcite was exploited to extract the exact age from social media posts. the flow chart of the methodology using bertweet is presented in figure 1 . the main steps of this approach are: ## 108 for the first methodology, the pre-trained llm 'australia/ mistral-7b-instruct-v0.2' was configured to reduce memory usage (potentially using 4-bit weights) by setting @xmath0.". the language model (llm) generates a response from the encoded conversation, with the maximum length of the response limited to 1000 tokens by setting @xmath1. additionally, enabling @xmath2, potentially yielding diverse outputs. for the second strategy, the textual data was pre-processed using the normalizetweet() function from the tweetnormalizer module. following tweet normalization, text data in both the training and validation sets undergo tokenization using the bertweet tokenizer (autotokenizer.from_pretrained(checkpoint)) with a maximum sequence length of 512 tokens. we optimized the model using the bertweet model's training parameters as experiment name: between-test, learning rate: 2e-5, number of training epochs: 5, checkpoint saving strategy: no checkpoints saved, and batch size per device as 64. the experiment used nvidia a100 gpu with python as the programming language. ## results the performance of the final trained models was assessed using the task 6 validation set prior to evaluating and submitting the prediction file on the test set. the performance of the models was evaluated using the f1-score for the positive class (i.e., posts annotated as "1"). the results of the validation set for mistral-7b-instruct-v0.2 (llm) and bertweet-base are reported in ## conclusion this work uses mistral-7b-instruct and bertweet models for precise age extraction from social media posts. the limitations of traditional approaches were overcome by directly inferring user age from text. evaluation of validation and test sets reveals significant performance disparities between the two models. bertweetbase outperforms mistral-7b-instruct with an f1-score of 0.90 for the positive class on the test set, showcasing its superior efficacy in age prediction. these results underscore the efficacy of advanced language models in enhancing demographic analysis and research applications. in future work, refining prompt engineering techniques and utilizing advanced models such as llama 3 can be exploited to enhance performance. additionally, exploring ensembling methods with models like bernie can offer even greater accuracy and robustness in demographic predictions.