Using BERT for Qualitative Content Analysis in Psychosocial Online Counseling (2020)

Abstract: Qualitative content analysis is a systematic method commonly used in the social sciences to analyze textual data from interviews or online discussions. However, this method usually requires high expertise and manual effort because human coders need to read, interpret, and manually annotate text passages. This is especially true if the system of categories used for annotation is complex and semantically rich. Therefore, qualitative content analysis could benefit greatly from automated coding. In this work, we investigate the usage of machine learning-based text classification models for automatic coding in the area of psycho-social online counseling. We developed a system of over 50 categories to analyze counseling conversations, labeled over 10.000 text passages manually, and evaluated the performance of different machine learning-based classifiers against human coders.

## introduction 1.psycho-social online counseling online counseling has developed into a fullfledged psycho-social counseling service in germany since the 1990s. today, people can get advice on a wide variety of psycho-social topics in web forums and dedicated text-based counseling platforms. online counseling is provided by psychosocial professionals who have received special training in this method. similar to face-to-face psycho-social counseling, some aspects are known to make up high-quality online counseling, but there is few empirical evidence for special impact factors @xcite . due to the complexity of the content, quantitative approaches have not been able to analyze the meaning and significance of methodical patterns in large numbers of consulting communications @xcite . it is, however, possible to understand and describe the meaning of online counseling content with qualitative approaches @xcite . this allows linking certain interventions of the counselors to the reactions of the clients on a caseby-case basis. but generalized statements on causal relationships are not possible with the small number of cases from qualitative studies @xcite ). an analysis of large numbers of counseling conversations using qualitative social research tools would help to better understand how successful online counseling works. few related studies on these topics are available. @xcite defined different models to measure general conversation strategies like adaptability, dealing with ambiguity, creativity, making progress or change in perspective and illustrated their applicability on a corpus of data from sms counseling. @xcite analyzed the quality of consulting communications based on video recordings. their automatic classifier used linguistic aspects of the content and could predict counseling quality with relatively good accuracy. however, neither of the mentioned approaches had the intention to recognize the meaning of individual phrases even though this deep understanding is crucial to eliminate weaknesses in the education of online counselors @xcite ). in addition, systems could be developed to provide online advisors with practical suggestions for improving their work. ## creating the data set online forums for psycho-social counseling provide a good basis for an empirical evaluation because they contain large amounts of publicly accessible data. for our study, we used posts from a german site for parent counseling. here, parents who have problems in bringing up their children are seeking advice. possible topics are, for example, drug abuse by the child or inadequate school performance. a user can start a new thread with a problem description. professional counselors reply and discuss solution approaches with the initial user and others. thus, each thread contains a series of posts with questions and suggestions about the initially described problem. since we are especially interested in counseling patterns, we focused on the posts of professional counselors in our analysis. ## model-based classification of psycho-social text sequences as a result of the created codebook and the collected data, our classification task consists of classifying psycho-social text sequences into one of 51 categories. for the training of the classifiers, the data set described in the previous section with 7169 samples is used. the created models are then evaluated against the 3072 samples in our test data set. ## bert vs. human coders coding of text passages is to some degree dependent on the subjective perception of the coders. especially for similar categories like "empathy" and "compassion", different coders will sometimes assign different labels to the same text. thus, even human coders which were trained on the usage of the codebook will not reach 100% agreement. to get a better understanding of the applicability of our model for automatic coding, we compared the coding performance of bert against a trained human coder familiar with the codebook ("expert") and an untrained human coder ("novice"). ## conclusion it has been shown that machine-based classifiers can reach human-like performance for the annotation of complex categories in psycho-social texts. the results indicate that the models learn to mimic the coding style of the initial creators of the training data. the trained bert model was even better in coding than a human novice. as in other areas of machine learning, this bears the risk that a model also learns the bias from the training data. therefore, it is important to understand and regularly check the decisions of the model by human experts. high coding quality could not be achieved for all codes, however. especially underrepresented categories, which are common in social sciences, are problematic. thus, a sufficient number of training samples is an obvious prerequisite for good results. the typical approach of social sciences in analyzing text corpora consists of coding one text after the other and ignoring unequal frequencies of the individual codes. our study shows that when using machine learning methods, it is better to generate training examples for as many categories as possible and pay less attention to the complete coding of individual texts. this is an important finding for the organization of future studies in this field. the investigation of misclassified sequences showed that many recorded misclassifications actually were minor mistakes. the model frequently chose not the actual but a very similar category such that even human experts would regard the assignment plausible. thus, codes with very similar meanings must be distinguished more sharply to give the model a chance to learn to differentiate. the analysis of the misclassified sequences of bert opens up new perspectives for the social sciences: more than half of the "incorrectly classified sequences" appeared to the human expert to be plausible or at least worthy of consideration. since the discussion of the understanding of individual text passages is an important element of social science research, such plausible misinterpretations can enrich the research process. they offer an alternative way of looking at reality and force the human coder to either rethink his assessments or to better justify them. currently, we are working on improving the classification performance. one approach is the generation of additional training data for underrepresented categories. another idea is using an ensemble of svm and bert as a classifier to better utilize the individual strengths of the different models. in any case, the findings on how the models work and perform help to consider such technical aspects in future social science research. with regard to the application domain, we can conclude that it is definitely possible to analyze online counseling conversations with the help of machine learning. we intend to use machine learning in future research projects to investigate correlations between the different techniques used by counselors and the characteristics and reactions of clients. in addition to the question of whether successful counselors use certain techniques significantly more often than others, it can now be clarified if certain approaches are particularly promising for certain target groups or specific problems. these findings can be integrated into the education of online counselors. furthermore, assistance systems are conceivable that support online counselors in real-time with information generated from this data. in any case, the results of this study have shown that it is possible to merge the advantages of qualitative and quantitative approaches in social science with the help of machine learning. automated data annotation for qualitative analysis is the cornerstone for future insights on an unprecedented level.