Overview of the CLP sych 2022 Shared Task: Capturing Moments of Change in Longitudinal User Posts (2022)

Abstract: We provide an overview of the CLPsych 2022 Shared Task, which focusses on the automatic identification of ‘Moments of Change’ in lon- gitudinal posts by individuals on social media and its connection with information regarding mental health . This year’s task introduced the notion of longitudinal modelling of the text generated by an individual online over time, along with appropriate temporally sen- sitive evaluation metrics. The Shared Task con- sisted of two subtasks: (a) the main task of cap- turing changes in an individual’s mood (dras- tic changes-‘Switches’- and gradual changes -‘Escalations’- on the basis of textual content shared online; and subsequently (b) the sub- task of identifying the suicide risk level of an individual – a continuation of the CLPsych 2019 Shared Task– where participants were encouraged to explore how the identification of changes in mood in task (a) can help with assessing suicidality risk in task (b).

## introduction increasingly the clinical community are looking for new and better diagnostic measures and tools for monitoring mental health conditions. over the past decade, there has been a surge in methods at the intersection of nlp and mental health, showing that signals for the diagnosis of certain conditions can be found in language. however, most research tasks have been defined on the basis of classifying individuals (e.g., on the basis of suicide risk @xcite or on the basis of having a mental health condition or not @xcite ), thus lacking the longitudinal aspect of monitoring an individual's mood and well-being in real-time. through this shared task we follow @xcite to introduce the problem of assessing changes in a person's mood over time on the basis {a.tsakalidis,m.liakata}@qmul.ac.uk of their linguistic content. for the purpose of the task we focus on posting activity in online social media platforms. in particular, given an individual's posts over a certain period in time, we aim: (a) at capturing those sub-periods during which an individual's mood deviates from their baseline mood -a post-level sequential classification task; (b) leveraging this task to help us assess the suicide risk level of the individual -a user-level classification task @xcite & a continuation of the 2019 shared task @xcite . thus, this year's shared task consists of two subtasks: (a) the main task of identifying mood changes in an individual's online posts over time and (b) assessing the suicide risk level of the invididual, where ideally participants will have been able to establish a connection between tasks a and b. this paper makes the following contributions: task b is a continuation of the work by @xcite and @xcite . given the posts of an individual, the aim is to classify their suicide risk into (a) no risk, (b) low, (c) moderate or (d) severe level. due to the very low number of users of (a) and (b) in our data, we have merged the no/low classes leading to a 3-label user classification task. participants were encouraged to use insights from task a in solving task b. ## dataset dataset creation for the two tasks ( §3.4) involved data collection & data relabelling ( §3.1), timeline extraction ( §3.2) and annotation ( §3.3). ## working in a secure environment the clpsych shared task 2021 @xcite was the first to be conducted in a secure environment to provide a high level of safety for sensitive data. we have also opted for carrying out this year's shared task in the same secure environment and continue efforts in protecting highly sensitive data. norc is an independent non-profit research institution at the university of chicago who provide the norc data enclave(r), chosen both this year and last for the shared task. compared to other solutions (see for instance @xcite ) the norc data enclave(r) (hereafter, 'de') does not rely on dedicated laptops but solely on a browser interface over https channels and citrix hdx technology, making the setup of a shared task more feasible. all teams (see §5) signed a data use agreement (dua) and terms and conditions (t&c) with norc before being provided with instructions to set up multi-factor authentication for login, procedures for requesting the ingress in the de of written code, libraries, models or additional data and procedures for technical support. all ingress of information into the de requires thorough system scans and human review to ensure the safety and integrity of the enclave. after login authorized users can access a secure virtual machine within the de. although all applications and data run on servers in the norc data center, the user interface is a familiar full windows 10 virtual desktop. the de is a closed environment: it does not have access to the internet and all functionalities for moving data in and out of the virtual space are disabled. this citrix-based technology is configured to prevent users from downloading output from the remote server to an external machine. similarly, other security protection features prevent the user from using the "cut and paste" feature in windows to move data from the citrix session into an excel spreadsheet residing on the local computer. in addition, the user is prevented from printing the data on a local computer. there is documentation regarding the virtual environment and how to securely connect to the dedicated de cluster on amazon web services (aws). to connect to the cluster (via ssh) users rely on putty and on the dedicated machine they can find a dedicated python 3.9.1 environment with all requested libraries available (see §5). users can both run code and submit batch jobs using the slurm cluster management while also monitoring the budget available for computational experiments. following last year's suggestions, we ensured participants would be able to use jupyter notebooks to implement code on the cluster through ssh tunneling and by opening the notebook in the browser of the windows machine. at the end of the shared task, each team was to inform norc to egress the predictions for the test set. due to an unprecedented technical issue out of norc's control, several teams faced issues with running their code a week prior to the system submissions deadlineto avoid eliminating the teams despite their continuous efforts throughout the shared task, we decided to distribute the data outside the de during the last few days on the basis of the signed dua. to ensure fairness, we asked all teams (i.e., not only the ones affected) to let us know if they would like to receive the data outside the enclave to help them with the system submission. we made it clear that those submitting their results within the de would feature separately in our evaluation (see tables 4 5 ), since they had more limited resources at their disposal. ## call for participation -teams selection we invited teams to register their interest in the shared task by providing details such as team members, motivation, related background, experience and nlp skills. we also asked for their requirements in terms of programming languages, libraries and pre-trained language models to prepare the set up in the de. given our limited resources pertaining to the functional costs of using the de, we were limited to accepting 15 teams (∼50 members) for participating in the shared task. therefore, we compiled a list of criteria that were given to two internal reviewers, along with the (anonymised) registrations of interest. the criteria were related to (a) the relevance of the team's background/current work to the shared task, (b) their motivation and likelihood of committing to the task and (c) details provided wrt technical requirements (see appendix c for the complete guidelines). based on the reviewers' assessments, we selected 13/37 teams to participate and asked another five applicant teams to be merged together into two groups, so as to accommodate as many requests as possible (one team was formed by three individual applicants, and another individual applicant was merged into a two-member team), leading to the acceptance of 18/37 requests (53 individuals). firstly, working on each timeline and label type independently, we calculate recall r (l) w (precision p (l) w ) by counting as "correct" a model prediction for label l if the prediction falls within a window of w posts around a post labelled as l in our ground truth -however, a post's predicted label can only be counted as 'correct' only once (at most). by increasing the value of w, we perform a less strict evaluation of a model. results are macro-averaged for each label independently across all timelines. secondly, we assess model performance on the basis of its ability to capture regions of change. for each true region r (l) gs within a timeline, we define its overlap o(r m as the intersection over union between the two sets. finally, we retrieve recall-and precisionbased coverage metrics (again, macro-averaged across all timelines for each label independently: ideally we want to see a system performing well on both window based and coverage metrics. task b. we use standard classification metrics (precision, recall and f1) for each user-based class label and macro-averaged. due to the low number of users in the 'low' class on the test set, we also report micro-averaged metrics; however, these are added for completeness purposes in our analysis (i.e., the teams were guided to improve their performance on a per-class and macro-average basis). ## shared task results this section outlines the submissions by each team. for task a, we also provide the results of three baselines: the majority classifier, a logistic regression (lr) trained on tfidf features, and bert trained using the focal loss on a related but separate dataset on the same task @xcite . for task b, we include the majority classifier and a lr trained on tfidf features from users' posts. ## conclusion we presented the overview of the clpsych 2022 shared task, focusing on (a) capturing changes in an individual's mood as self-disclosed online and (b) classifying the individual's suicide risk levelas well as studying the link between the two tasks. the best results for (a) showcase the importance of taking into account the sequence-aware modelling of an individual's online shared content, whereas the link between the two tasks has been highlighted on the basis of the best results achieved for (b). following last year's setting @xcite , we utilised norc's enclave. faced with challenges out of our and norc's control, we pro-vide directions for shared tasks on sensitive domains ( §9). our aim for the future is to emphasize the need for research on longitudinal tracking and modelling of a user's mental health, under a common experimental setting in a secure environment.