J ohn_ S now_ L abs@ SMM 4 H ’22: Social Media Mining for Health (# SMM 4 H ) with Spark NLP (2022)

Abstract: Social media has become a major source of information for healthcare professionals but due to the growing volume of data in unstructured format, analyzing these resources accurately has become a challenge. In this study, we trained health related NER and classification models on different datasets published within the Social Media Mining for Health Applications (#SMM4H 2022) workshop. Transformer based Bert for Token Classification and Bert for Sequence Classification algorithms as well as vanilla NER and text classification algorithms from Spark NLP library were utilized during this study without changing the underlying DL architecture. The trained models are available within a production-grade code base as part of the Spark NLP library; can scale up for training and inference in any Spark cluster; has GPU support and libraries for popular programming languages such as Python, R, Scala and Java.

## introduction a primary building block in text mining systems is named entity recognition (ner) -which is regarded as a critical precursor for question answering, topic modeling, information retrieval, etc. @xcite . in the medical domain, ner recognizes the first meaningful chunks out of a clinical note, which are then fed down the processing pipeline as an input to subsequent downstream tasks such as clinical assertion status detection @xcite , clinical entity resolution @xcite and deidentification of sensitive data @xcite . text classification is also one of the important tasks of machine learning and has been extensively used in several areas of nlp. it can be defined as assigning a sentence or document an appropriate category. in this paper, we share our results on the #smm4h shared task, which involves nlp challenges of using social media data for health research, including informal, colloquial expressions and misspellings of clinical concepts, noise, data sparsity, ambiguity, and multilingual posts. the contest involves classification and ner tasks. as john snow labs team, we contributed to many tasks and trained several ner and classification models on english and spanish social media data. ## implementation details models for classification tasks were trained by using bert for sequence classifier (bfsc) and bert for token classifier (btfc) was used for training ner tasks. embeddings and hyper parameters used for training the documents are shown in ## experimental results using the official training sets from the contest, we trained models and obtained metrics on the test sets used in the challenges. the results are shown in unless otherwise specified, biobert-basecased-v1.2 embeddings were used for model training of english texts. often times, cleaning the social media material (removing emojis, tags, repeating punctuations etc.) did not improve the results. task 1a is about classifying tweets reporting adverse drug events (ades). bfsc gave the best results. the training dataset was unbalanced; therefore, it has been enriched with different ade datasets @xcite to increase the number of ade entities. task 1b involves detecting ade intervals in tweets. the model was trained using bftc. as mentioned above, the training dataset was unbalanced and just like task 1a, the dataset was enriched by using datasets @xcite with the aim of increasing the number of ade entities. in task 2a, classification of stance in tweets about health mandates is expected. bfsc with biobert-base-cased-v1.2 embeddings produced the best results, whereas bert_base_cased embeddings produced slightly higher metrics. task 2b involves binary classification of premise in tweets about health mandates. bfsc produced the better f1 score values. in task 3a, detection of tweets where the users self-declare changing their medication treatments is expected. the model trained using bfsc produced higher f1 score values. training data was unbalanced and using different embeddings and cleaning the tweets did not improve the results. task 3b involves binary classification of drug reviews from webmd.com. bfsc produced around 0.84 f1 score values. task 4 involves identification of the self-report exact age in tweet posts. bfsc produced the best results (f1-score of 0.82). for task 5, all the tested configurations obtained decent accuracies over the validation dataset. the difference of scores between the two classes is probably due to the imbalance in the dataset. task 6 involves identification of self-reported covid-19 vaccination status in english tweets. the two classes in the provided training dataset are vaccine_chatter and self_reports (tweets of users clearly stating that they have been vaccinated). the classes are significantly imbalanced with a ratio of 1 to 8. bfsc produced an f1 score value (for the positive class) of 0.72. task 7 involves the identification of selfreported intimate partner violence (ipv) in english tweets. the dataset is significantly unbalanced, and the negative tweets include non-ipv domestic violence and non-self-reported ipv, which can hardly be distinguished. where the model could not select between domestic violence and ipv, better results were obtained by introducing family-related words (father, mother, daughter, sister etc.) to the model and enabling it to learn about domestic violence. bfsc produced f1 values around 0.72 (for the positive class) for the validation dataset. task 8 is about detecting tweets that are selfdisclosures of chronic stress. for the classification task, a pre-trained roberta for sequence classification model from hugging face with roberta embeddings is fine-tuned on the given dataset. without cleaning the tweets, the model resulted in better scores. the validation f1 score was 0.78 and the test f1 score is 0.76. task 9 involves identification of the exact age in social media forum (reddit) posts. bfsc produced the best results (f1-score of 0.87). task 10 requires detection of disease mentions in spanish tweets, and bftc was used to produce the models. huggingface's 5-language embeddings (bert-base-5lang-cased) were used to get the highest f1 score values. ## conclusion in this study, we show through experiments on #smm4h contest datasets in two languages that models trained by bfsc and bftc can easily be adapted to the spark nlp environment and achieve decent scores on social/health-related datasets with zero code changes. we trained bfsc models for classification and bftc models for ner purposes that can be used as a pretrained model out of the box within spark nlp for healthcare library. considering the f1 scores, all the models produced decent performances during training. the problem of getting low metrics due to unbalanced datasets was solved by enriching the dataset with data from similar datasets and consequently, there was a substantial increase in the f1 scores.