Overview of the Seventh Social Media Mining for Health Applications (# SMM 4 H ) Shared Tasks at COLING 2022 (2022)

Abstract: For the past seven years, the Social Media Mining for Health Applications (#SMM4H) shared tasks have promoted the community-driven development and evaluation of advanced natural language processing systems to detect, extract, and normalize health-related information in public, user-generated content. This seventh iteration consists of ten tasks that include English and Spanish posts on Twitter, Reddit, and WebMD. Interest in the #SMM4H shared tasks continues to grow, with 117 teams that registered and 54 teams that participated in at least one task—a 17.5% and 35% increase in registration and participation, respectively, over the last iteration. This paper provides an overview of the tasks and participants’ systems. The data sets remain available upon request, and new systems can be evaluated through the post-evaluation phase on CodaLab.

## introduction for the past seven years, the social media mining for health (#smm4h) workshop has been a competitive platform to promote the development and evaluation of advanced natural language processing systems to detect, extract, and normalize healthrelated information in user generated content publicly available online. for the seventh iteration of the workshop shared tasks, researchers from international institutions challenged the community with ten tasks. the tasks run on various media (tweets, reviews and reddit posts) and languages (english and spanish): classification, detection and normalization of adverse events mentions in tweets (task 1), classification of stance and premise in tweets about health mandates related to covid-19 (task 2), classification of changes in medication treatments in tweets and webmd reviews (task 3), classification of tweets self-reporting exact age (task 4), classification of tweets containing self-reported covid-19 symptoms -in spanish (task 5), classification of tweets which indicate self-reported covid-19 vaccination status (task 6), classification of self-reported intimate partner violence on twitter (task 7), classification of selfreported chronic stress on twitter (task 8), classification of reddit posts self-reporting exact age (task 9), detection of disease mentions in tweetsin spanish (task 10). this iteration shows that the interest of the community for the shared tasks continues to grow with 117 teams registered, representing a 17.5% growth compare to the last iteration in 2021. the growth of interest translated in an increase of participation with 54 teams having submitted their predictions for at least one task, a growth of 35% compared to last year participation. while all task organizers were free to change the modality for their task, we recommended generic guidelines to organize the tasks. we allowed all participants to register in one or more tasks. upon acceptance, we provided the participants with a training and a validation set for each task they registered in during the practice period. we released unlabeled test sets at the beginning of the evaluation period of the competition, a period which was 4 days long. during this period, each team could upload up to 3 predictions sets for the labels of the test sets to the web-based platform, codalab. codalab automatically evaluated their performance. we report in this overview the results achieved with the best set of three predictions sets submitted. we left the post-evaluation period opened indefinitely on codalab for all tasks run during this iteration. interested readers can request the datasets and upload their own predictions to the codalab server to compare their performance with the winners of each task. in section 2, we briefly describe and motivate the ten tasks of the competition. in section 3, we present the performance and a short interpretation of the results for each task. in appendix 4, we provide the list of the publications describing the systems of the competing teams along with the team ids referring them in section 3. for task 1, participants were asked to develop methods to extract adverse drug effects (ade) from tweets containing drug mentions. such automated methods are considered necessary for social media pharmacovigilance efforts for monitoring emergence of ades in post-market surveillance of drugs, especially in vulnerable populations such as children, pregnant women and the elderly who are often excluded from clinical trials. reliable signals generated from such social media pharmacovigilance pipelines can be complementary to traditional pharmacovigilance efforts such as voluntary consumer and health provider reporting. ade mining in tweets has been the longest running tasks at the smm4h shared task. task 1 presents three subtasks in increasing order of complexity: (task 1a) identify tweets that contain one or more ades, (task 1b) in addition to task 1a, extract the text span of reported ades in tweets, and (task 1c) in addition to tasks 1a and 1b, normalize extracted spans to meddra ontology's preferred terms https://meddra.org/ . for this task, we used the same dataset as the previous year. dur-ing the training state, the dataset contains a total of 18,300 tweets with 17,385 tweets for training and 915 tweets for validation @xcite . during the evaluation stage, we performed a blind evaluation on 10,984 tweets. the tweets were manually annotated by experts with: (a) binary labels of hasade and noade indicating presence of one or more ades, (b) starting and ending indices of the ade mention(s) in the text, and (c) the normalized meddra lower-level term (llt) that were evaluated at the higher preferred term (pt) level. med-dra contains about 79,000 llt terms and about 23,000 preferred terms, making it important for the developed systems to be capable of identifying and normalizing ades that were not occurring in the training set. task 1 presents multiple technical challenges such as class imbalance and normalizing into a large potential label space. submissions were evaluated and ranked based on the f 1 -score for the ade class for subtask 1a, f 1 -score for overlapping ade mentions for subtask 1b, and f 1 -score for overlapping ade mentions with matching preferred term ids for subtask 1c. task 1 is hosted on codalab at https://codalab.lisn. upsaclay.fr/competitions/2073 . 2.2 task 2: classification of stance and premise in tweets about health mandates related to covid-19 (in english) for task 2, we focus on argument mining (or argumentation mining) for extracting arguments from covid-related tweets. tweets express views towards three claims/topics associated with governments' restrictions: (i) keeping schools closed, (ii) stay-at-home orders, and (iii) wearing a face mask. systems for detecting people's stances and their premises about governments' health mandates related to covid-19 can help to gauge the level of cooperation in society which is essential for stopping the spread of the virus. the task consists of two sub-tasks: (i) task 2a on stance detection, and (ii) task 2b on premise classification. task 2a: stance detection the first sub-task aims to determine the point of view (stance) of the text's author in relation to the given claim. the tweets are manually annotated for stance according to three categories: in-favor, against, and neither. task 2b: premise classification the second sub-task is to predict whether at least one premise/argument is mentioned. a given tweet is considered as having a premise if it contains set stance classes premise classes in-favor against neither 1 0 train 1346 874 1336 1331 2225 valid 244 158 198 221 379 test 526 570 904 716 1284 a statement that can be used as an argument in a discussion. for instance, the annotator could use it to convince an opponent about the given claim. for example, both tweets "petition to stop calling people who don't wear masks "anti-maskers". instead, let's just call them terrorists. #coronavirus" and "masks help prevent the spread of the disease. please, #wearamask" are in favor of the claim, yet only the second tweet contains the argument. the tweets were manually annotated for binary classification, and participants of this subtask were required to submit whether each tweet has a premise (1) or not (0). we split an existing corpus of 4,269 covidrelated tweets @xcite into a training set of 3,669 tweets and a validation set of 600 tweets. this corpus includes annotations for task 2a. we added a new annotation layer for task 2b to this corpus. in order to create the test set, we collected new tweets using 33 keywords such as #openschools, #lockdownnow, #nomasks. we removed the hashtags from the tweets to exclude obvious signals (e.g., #saynotomasks can be seen as the "against" hashtag). the test set for task 2a and all three sets for task 2b were manually annotated. three yandex.toloka 1 annotators' crowdsourced labels were aggregated into a single label @xcite . in task 3, we challenged the participants to design binary classifiers to detect posts where social media users self-declare changing their medication treatments, regardless of being advised by a health care professional to do so. such changes are, for example, not filling a prescription, stopping a treatment, changing a dosage, forgetting to take the drugs, etc. this is an important task since it is the first step toward detecting patients who may be non-adherent to their treatments and it would allow us to further understand their reasons when they are expressed on social media @xcite . we released two corpora for the task, 9,830 tweets from twitter and 12,972 drug reviews from webmd, a website which provides an opportunity for users to comment anonymously on their personal experience with a drug in a free text form. two annotators labeled the posts with "1" when the posts state a change in medication regimen, the positive examples, "0" otherwise, the negative examples. the inter-annotator agreements were moderate on both corpora with 0.65 and 0.74 cohen kappa scores on the corpus of tweets and reviews, respectively. with 7,222 positive examples and 5,750 negative examples, the corpus of webmd is naturally balanced with an imbalance ratio of 0.80. the corpus of tweets is more challenging for training classifiers with supervision, which is the default approach to solve the task. the corpus of tweets has a strong imbalance with only 864 positive examples for 8,966 negative examples, that is a 10.38 imbalance ratio. we split randomly each corpus into a training, a validation, and a test set with 90%/10%/10% of the total reviews available and 60%/16%/24% of the total tweets available in each set. during the practice period of the competition, we provided the participants the training and validation sets. during the evaluation period, all participants had 5 days to compute their predictions on the test set and submit them on codalab for evaluation. we added 11,835 reviews and 13,000 tweets as decoys in the test sets to prevent manual correction of the predictions. we evaluated participants' systems with the precision, recall and f 1 -score for the positive class, that is tweets or reviews mentioning a change in medication treatments. the task 3 is hosted on codalab at https://codalab.lisn. upsaclay.fr/competitions/2138 . 2.4 task 4: classification of tweets self-reporting exact age (in english) a limitation of using twitter data for research applications is that users may not be representative of the general population. therefore, advancing the utility of twitter data for research applications requires methods for automatically detecting demographic information, including users' age. automatically identifying the exact age of twitter users, rather than their age groups, would enable the large-scale use of twitter data for applications that do not align with the predefined age groupings of extant models, including health applications such as identifying specific age-related risk factors for observational studies (golder et al., 2019), or selecting age-based study populations (davies et al., 2022). as a first step, this binary classification task involves automatically distinguishing tweets that self-report the user's exact age ("age" tweets) from those that do not ("no age" tweets). the training set contains 8800 annotated tweets: 2834 (32%) "age" tweets and 5966 (68%) "no age" tweets. the validation set contains 2200 annotated tweets: 709 (32%) "age" tweets and 1491 (68%) "no age" tweets. the test set also contains 2200 annotated tweets: 768 (35%) "age" tweets and 1432 (65%) "no age" tweets. inter-annotator agreement (fleiss' kappa), based on 1000 tweets annotated by five annotators, was 0.80. the task 4 is hosted on codalab at https://codalab.lisn. upsaclay.fr/competitions/3566 . ## conclusion this paper presented an overview of the #smm4h 2022 shared tasks. with ten tasks proposed this year, the interest and the participation in the #smm4h shared tasks continues to grow. all best