Truth in the Noise: Unveiling Authentic Dementia Self-Disclosure Statements in Social Media with LLM s (2024)

Abstract: Identifying self-disclosed health diagnoses in social media data using regular expressions (e.g. “I’ve been diagnosed with <Disease X>”) is a well-established approach for creating ad hoc cohorts of individuals with specific health conditions. However there is evidence to suggest that this method of identifying individuals is unreliable when creating cohorts for some mental health and neurodegenerative conditions. In the case of dementia, the focus of this paper, diagnostic disclosures are frequently whimsical or sardonic, rather than indicative of an authentic diagnosis or underlying disease state (e.g. “I forgot my keys again. I’ve got dementia!”). With this work and utilising an annotated corpus of 14,025 dementia diagnostic self-disclosure posts derived from Twitter, we leveraged LLMs to distinguish between “authentic” dementia self-disclosures and “inauthentic” self-disclosures. Specifically, we implemented a genetic algorithm that evolves prompts using various state-of-the-art prompt engineering techniques, including chain of thought, self-critique, generated knowledge, and expert prompting. Our results showed that, of the methods tested, the evolved self-critique prompt engineering method achieved the best result, with an F1-score of 0.8.

## introduction longitudinal changes in linguistic abilities have been studied to identify a relationship between language decline and the onset of dementia @xcite . the nun study, a longitudinal investigation into alzheimer's disease, examined this relationship @xcite . @xcite in their study that higher linguistic abilities in early adulthood, measured by the proportion of complex sentences in writing samples, were linked to a lower risk of developing dementia. while longitudinal research offers valuable insights into causal relationships, it is often challenging and costly to collect such data (m. @xcite . social media data has become a promising source for creating cohorts for longitudinal studies @xcite , as data can be continuously and passively collected from users' interactions over extended periods. a further significant advantage of social media data is that each post is timestamped, making it easy to track changes over time. this allows researchers to analyze linguistic patterns with precise temporal context, capturing everyday language use across various contexts. this characteristic enhances the ability to study longitudinal changes in language and its relation to conditions such as dementia @xcite . a key step in social media analysis, following the collection of user data, is the annotation process @xcite . accurate annotation is vital, as correctly labelling users enables researchers to distinguish between groups and analyze their differences. while methods relying solely on pattern matching for the identification of self-disclosure statements are straightforward to implement, they often prove unreliable in the context of mental health and neurodegenerative condition due to the tendency of such disclosures to be humorous, whimsical, or sardonic. in this research, we leverage large language models (llms) to automate the annotation of social media data related to dementia self-disclosure. llm performance is highly dependent on the quality of the prompts guiding the model. to optimize these prompts, we implemented a genetic algorithm that evolves them using various state-ofthe-art (sota) prompt engineering techniques. by monitoring the performance of these techniques, we gained valuable insights into which methods are most effective for this task. our prompts were also designed as detailed guidelines, enabling the model to detect subtle linguistic patterns critical to identifying authentic dementia-related disclosures. this approach not only improves annotation accuracy but also enhances interpretability, offering researchers insights into the linguistic features of dementia self-disclosure on social media. ## method 3.1 data collection to construct our dataset, we used the twitter academic api to collect tweets containing search terms like "i have dementia," yielding a total of 14,025 tweets. the data collection took place between october and november 2022. for each self-disclosure tweet, we also gathered the five posts immediately preceding and following the self-disclosure to assess their context. for the complete list of selfdisclosure terms used for the data collection, please refer to appendix a. three authors of the paper were responsible for annotating the dataset. to improve inter-annotator agreement, they completed four annotation blocks, each consisting of 1,991 tweets. a substantial inter-annotator agreement was achieved, with a pairwise cohen's kappa of 0.68 @xcite . of the tweets collected using the search terms, less than 20% were authentic. from the remaining data we built a balanced dataset with a 50/50 distribution of authentic and inauthentic statements by applying upsampling. the dataset was divided into stratified training and testing sets, following an 80/20 split. the training and testing datasets were verified to ensure there was no crosscontamination between them. the training dataset was then divided into 10 stratified batches. ## results and discussion the optimized prompt (refer to appendix e) achieved an accuracy of 0.8 and an f1-score of 0.8, outperforming the bert classifier, which obtained an accuracy of 0.7 and an f1-score of 0.71. in figure 2 , the distribution of mutation types among individuals across generations is illustrated. the most prevalent mutation type observed throughout multiple generations stemmed from the sc prompt engineering technique, with the topperforming prompt from the final generation being a product of a sc mutation prompt. however, upon reviewing the prompt, we observed the integration of elements from various prompt engineering methods. prompts derived from gk typically include an enumeration of components to evaluate. when followed by an sc mutation, the prompt addresses shortcomings in the components suggested and guides the model to contextualize them properly. additionally, elements of cot mutations are evident in the logical step-by-step structure of the prompt. all these characteristics were present in the optimized prompt. therefore, our findings suggest that the optimal prompt engineering approach involves a blend of different techniques. the adaptive prompt engineering technique was developed and evaluated using an open-access model that can be run locally, enabling researchers to analyze sensitive content without needing to send it to third-party organizations. additionally, since the model is open-access, there are no associ-ated usage fees, which reduces costs and improves accessibility, particularly in less well-resourced settings. our algorithm also offers an accessible approach for public health researchers to identify self-diagnosed patients on social media for cohort building. it minimizes the need for expertise in machine learning or prompt engineering, as sota techniques are integrated into the algorithm. moreover, our algorithm allows for upgrades upon the discovery of new prompt engineering techniques, requiring only their addition to the mutation pool. ## conclusion we used a genetic algorithm to optimize prompts for llms to detect self-disclosed dementia statements in tweets. the optimal prompt achieved an accuracy of 0.8 and an f1 score of 0.8, surpassing the bert classifier, which had an accuracy of 0.7 and an f1 score of 0.71. additionally, it significantly outperformed a method that would solely rely on key search terms to label users as having dementia, as our annotation process revealed that less than 20% of the collected tweets with dementia self-disclosure statements were authentic. the algorithm used sota prompt engineering methods, and analysis revealed that sc mutations outperformed the other mutation types. although our algorithm was designed to automate the annotation of dementia-related data, it can also assist in the annotation of other types of data when provided with the appropriate datasets. we envision that by adapting our algorithm, researchers may find it helpful in supporting the annotation process across various domains, improving efficiency and reducing manual labor.