Conversation-Aware Filtering of Online Patient Forum Messages (2020)

Abstract: Previous approaches to NLP tasks on online patient forums have been limited to single posts as units, thereby neglecting the overarching conversational structure. In this paper we explore the benefit of exploiting conversational context for filtering posts relevant to a specific medical topic. We experiment with two approaches to add conversational context to a BERT model: a sequential CRF layer and manually engineered features. Although neither approach can outperform the F1 score of the BERT baseline, we find that adding a sequential layer improves precision for all target classes whereas adding a non-sequential layer with manually engineered features leads to a higher recall for two out of three target classes. Thus, depending on the end goal, conversation-aware modelling may be beneficial for identifying relevant messages. We hope our findings encourage other researchers in this domain to move beyond studying messages in isolation towards more discourse-based data collection and classification. We release our code for the purpose of follow-up research.

## introduction in the past decade, social media has emerged as a source of valuable knowledge in the health domain @xcite , for instance during the covid-19 pandemic @xcite ) @xcite . in order to use social media to answer a medical question, it is necessary to identify posts on the forum that are relevant to the question at hand e.g. posts mentioning adverse drug responses (adrs) @xcite , personal experiences @xcite , medication abuse @xcite or medical misinformation @xcite . this filtering step is often the first step of the analysis pipeline. in this paper, we will refer to this specific type of filtering as relevance classification. previous automatic methods for medical relevance classification generally consider posts as units without context, thereby ignoring any information that can be gained from conversational context. one example of such an approach is the recent shared task on adr relevance classification @xcite . yet, including the conversational context may prove beneficial to relevance classification, as responses in a thread often relate to previous responses. for example, responses to a question or comment about a specific side effect are likely to also concern this side effect. to test this hypothesis, we investigate how positive labels are distributed across and within conversational threads. at present, only one study into medical relevance classification has included some engineered features to capture aspects of the conversational structure @xcite . however, as this study includes only two discourse-based features, the effect of including manually engineered features that capture conversational structure is still largely unknown for relevance classification tasks. furthermore, including the relation between posts on a discourse level may also be able to improve classifier performance. each post serves a conversational function in a dialogue, e.g. a question, explanation or statement @xcite . these functions are called dialogue acts @xcite . we have not found any study that included dialogue acts as features for medical relevance classification. as an alternative to using manually engineered features, conversational threads can also be modelled with a sequential model. this has proven beneficial in other fields such as rumor classification in social media discussions @xcite . as of yet, the use of sequential models for medical relevance classification has also not been explored. we address the following research questions in this paper: ## related work the use of conversational structure for improving the performance of classifiers of social media posts is prevalent in the field of rumor classification @xcite and related fields like disagreement detection @xcite . conversational structure has previously been exploited through (a) manually engineered features or (b) sequential classifiers. the most commonly employed engineered features to model the conversational structure are the similarity to the previous message and to the thread in general @xcite . in addition to these features, the current state-of-the-art model on a leading shared task for rumor stance classification (rumoureval-2019) uses the label of the previous message and the distance to the start of the thread @xcite . in the health domain, the only study that employs manually engineered features for conversational structure is @xcite . specifically, they use the running count of positive labels and the distance to the previous positive label. in this study, we will employ the above features as well as expand upon them with additional discourse-related features. other studies have used sequential classifiers to model the discursive nature of social media, although according to @xcite this is "still in its infancy" (p. 276). their comparison of various classifiers for rumor stance classification revealed that sequential classifiers outperform non-sequential classifiers overall. this is probably due to their ability to leverage information about sequential structure and preceding labels. furthermore, @xcite found that sequential classifiers did not benefit from contextual features representing thread context (e.g. similarity to the source tweet) whereas nonsequential classifiers did. they speculate that sequential classifiers take the surrounding context into account implicitly. to see if this also holds true for relevance classification in medical social media, we will compare the addition of conversation-aware features to both sequential and non-sequential models. ## discussion we find that the distribution of target posts across discussion threads is skewed and that within a conversational thread posts cluster together. thus, our hypothesis that the probability of a target post occurring is related to the conversational structure appears valid. in answer to rq1, we find that a sequential crf layer on top of a bert model improves precision slightly, although only significantly so for adr detection. in answer to rq2, we find that the addition of manually engineered features representing thread context often does not aid performance. the one consistent exception is when combined with a non-sequential svm layer on top of a bert model. this combination can improve recall for all target classes, although not significantly. an additional layer on top of a bert model that is able to capture the thread context appears to offer flexibility in tailoring the model towards a higher recall or precision. in future work, we plan to investigate the benefit of including conversational context for other tasks such as concept normalization of adr. for all the data sets included in this study, a pre-selection of discussion threads was made prior to annotation to ensure a higher proportion of target posts. we expect that both sequential models and manually engineered features of thread context may prove more beneficial when such a pre-selection does not take place and the target class is even more imbalanced. thus, our results may be an underestimation of the benefit of conversational context for finding 'needles in the haystack'. finally, our findings call into question the practice of splitting data into folds without taking the discussion context into account. in this study, we split the folds per discussion thread and we recommend others to consider doing so when dealing with multiple posts from the same thread, as neglecting to do so when there are dependencies between posts may bias model performance. this is especially important when threads contain duplicate posts.