UKYNLP @ SMM 4 H 2024: Language Model Methods for Health Entity Tagging and Classification on Social Media (Tasks 4 & 5) (2024)

Abstract: We describe the methods and results of our submission to the 9th Social Media Mining for Health Research and Applications (SMM4H) 2024 shared tasks 4 and 5. Task 4 involved extracting the clinical and social impacts of non-medical substance use and task 5 focused on the binary classification of tweets reporting children’s medical disorders. We employed encoder language models and their ensembles, achieving the top score on task 4 and a high score for task 5.

## introduction in today's digital landscape, social media platforms have evolved beyond mere communication channels, transforming into vital sources of real-time, user-generated data that reflect a wide array of public experiences and concerns. this transformation is particularly pertinent in the realm of public health, where social media discussions provide invaluable insights into both prevalent health issues and the personal impacts of various conditions. the critical analysis of these online dialogues is essential for understanding and addressing two significant public health challenges: nonmedical substance use and children's health disorders. task 4 and task 5, conducted as part of the 9th social media mining for health research and applications (smm4h) workshop, illustrate the potential of leveraging social media for health research. task 4 focuses on extracting the clinical and social impacts of nonmedical substance use from reddit discussions. understanding these impacts is vital for developing nuanced interventions and educational programs aimed at mitigating the adverse outcomes of substance misuse. the analysis of user discussions can reveal the multifaceted consequences of such behavior, informing targeted interventions and effective medication strategies. task 5 addresses the binary classification of tweets related to children's health disorders, distinguishing between tweets from users reporting a child with disorders such as adhd, asd, delayed speech, or asthma from those that merely mention these conditions without indicating a personal effect. this task underlines the necessity for innovative data collection methods that can complement traditional epidemiological approaches, which often face logistical and financial barriers. by analyzing twitter data, researchers can access a broader dataset, uncovering patterns and insights with speed and scale infeasible in conventional studies. this capability is crucial for developing targeted public health interventions and building support services that meet the needs of families dealing with children's health disorders. the task 4 dataset consists of 26,126 reddit posts (60% for training, 20% for validation, and 20% for testing/evaluation). only 318 posts (approximately 1.22%) were annotated for clinical impacts (health, physical condition, and mental well-being) or social impacts (effects on social relationships, community dynamics, and broader societal issues) @xcite . thus this dataset has high imbalance in the sense that most posts would not contain any task specific entities. ## methodology for task 4, the main nlp task is named entity recognition (ner), for which we use a span-based encoder-only model that enumerates contiguous spans of tokens up to a certain length (we used 8) and classifies each of them as any of the allowed entity types. we used the princeton university relation extraction (pure) @xcite pipeline's entity model component, which is originally inspired by other prior efforts @xcite . span-level representations for each token are first derived from pre-trained language models, such as the bert base uncased model @xcite and the albert xx-large model @xcite , both english models. the span representation is the concatenation of these encoder-only model outputs for the first and last token embeddings along with an embedding for span length. a feed forward network processes these span representations to compute the probability distribution of entity types. as a variation, we also incorporated a bidirectional lstm (bilstm) layer, with 150 units in each direction (totaling 300 units), between the span embeddings and the classification layers in the bert and albert models; this was shown to improve results in prior experiments and was also reported by @xcite . the dataset for task 4 was used with varying batch sizes and context windows. here, a context window denotes the extent of textual context surrounding the target sentence that is being considered during the entity extraction process. we did some basic preprocessing of the messages such as lower casing and converting it into the format expected by our span based approach. for hyperparameters used in task 4, please refer to for task 5, we fine-tuned pretrained encoderonly language models for tweet classification. specifically, we used a deberta v2 xlarge model @xcite , a deberta v3 large model, already fine-tuned on the multi-nli (mnli) dataset @xcite , and a roberta @xcite base model, previously fine-tuned on a general tweet dataset @xcite . all models were further fine-tuned on the task 5 dataset, which focuses on children's disorders such as adhd, asd, speech delays, and asthma. the best-performing models were obtained by systematically adjusting key hyperparameters such as the number of epochs, batch size, and learning rate. the specific hyperparameters used for the models are detailed in ## concluding remarks in this report, we reviewed our approaches to smm4h 2024 tasks 4 and 5, which mostly focused on applying encoder-only language models to ner and text classification. we were officially informed of our first place in task 4 results and it appears our post-evaluation scores for task 5 are near the top. we conducted preliminary error analyses that mostly pointed to informal/casual language (that does not adhere to grammatical norms) as a prominent trait characterizing both false positives and false negatives. while the precision and recall are around the same range for task 5, for the ner task 4 we notice recall is at least ten points higher than precision; the difference was close to 12 points in row 2 ( we did not attempt to use decoder-only large language models (llms) (e.g., mistral and llama) because in our lab's prior explorations @xcite , with ample training data, encoder-only models always fared better for information extraction (ie) tasks. this was also observed by other researchers who focused on ie tasks that need language understanding capabilities more than gener-ative skills. however, it is worth reassessing this with bigger llms (7b and more parameters) to see if they excel at supervised ner and classification tasks. our working hypothesis is that llms may show benefits when dealing with shorter entity ner tasks (single or two token entities) but encoder-only models might still be the best option for handling longer entities that were more common in task 4. additionally, we also plan to exploit even bigger llms (e.g, gpt-4) to generate synthetic examples that augment training data for both tasks to see if that benefits the overall performance. however, it is not clear how augmentation can be carried out for ner to retain longer named entities intact while changing the overall sentence with decent coherence and preserved meaning. effective augmentation for classification may be relatively easier to achieve. these are some future directions we hope to pursue soon.