Predicting independent living outcomes from written reports of social workers (2020)

Abstract: In social care environments, the main goal of social workers is to foster independent living by their clients. An important task is thus to monitor progress towards reaching independence in different areas of their patients’ life. To support this task, we present an approach that extracts indications of independence on different life aspects from the day-to-day documentation that social workers create. We describe the process of collecting and annotating a corresponding corpus created from data records of two social work institutions with a focus on disability care. We show that the agreement on the task of annotating the observations of social workers with respect to discrete independent levels yields a high agreement of .74 as measured by Fleiss’ Kappa. We present a classification approach towards automatically classifying an observation into the discrete independence levels and present results for different types of classifiers. Against our original expectation, we show that we reach F-Measures (macro) of 95% averaged across topics, showing that this task can be automatically solved.

## introduction social workers are concerned with improving abilities and increasing confidence of their clients with the goal of supporting them in living their lives independently. a traditional taxonomy, called metzler's taxonomy 1 @xcite , considers the following life categories in which independence is to be reached: 'everyday life', 'individual basic care', 'relationships', 'participation in cultural and social life', 'communica-1 https://www.soziales.niedersachsen.de/startseite/behinder te menschen/eingliederungshilfe behinderte menschen/bedarffeststellung-des-bedarfs-gruppen-fuer-leistungsberechtigtemit-vergleichbarem-bedarf-94870.html (as consulted online 11.10.2020), there, the taxonomy can be found in appendix 2.4 tion', 'emotional and psychological development' and 'health promotion and maintenance'. each category has specific subcategories, with overall 34 categories. social workers document their observations with respect to the behaviour of clients in day-to-day records that capture the whole trajectory of their clients. interviews carried out with social workers have revealed that they would profit substantially from automatic summarization of the trajectories of patients, in particular their progress over time on reaching independence in the different categories defined by @xcite towards this goal, in this paper we investigate whether it is possible to automatize this task. we frame the task as a classification problem in which each observation in the records about the patient is classified into a discrete independence level. in particular, we distinguish the following five independence levels: 1 ('able to accomplish a certain task alone'), 2 ('able to accomplish a given task with help'), 3 ('partly able to accomplish a task with help'), 4 ('unable to accomplish a certain task'). the neutral label 5 applies for documentations that do not allow a conclusion regarding the level of independence. in as one contribution of this paper, we describe the process of collecting a corpus on the basis of data from two social care institutions. the corpus has been annotated by social work students that for our purposes can be regarded as domain experts. after several annotation rounds, the annotator agreement reached 0.74 as measured by fleiss' kappa, corresponding to a substantial agreement. building on this corpus, we train different classifiers on the annotated data and show that the classifiers can reach a macro f-measure of 95% averaged across the 34 metzler categories on the task of predicting independence levels. the best results overall are obtained with a linear svm (one-vs-one) classifier. the best result is achieved on metzler's categories m-6 ('managing money'), m-11 ('get up / go to bed'), m-17 ('organization of free time / private activities'), m-19 ('encounters in social groups'), m-22 ('compensation of sensory impairments and communication disorders'), m-24 and m-25 on orientation in familiar and unfamiliar surroundings, m-27 and m-28 on coping with psychological disorders (100% fmeasure), while the worst result is achieved on metzler's category m-21 ('development of future perspectives, life planning') (92% f-measure). we rely on a feature ablation experiment to determine the importance of different features on the task. the paper is structured as follows: in section 2, we discuss related work for the problem of predicting extra-linguistic personal attributes. in section 3, we describe how the data was obtained from two different disability care institutions. in section 4, the method is described, including the development of guidelines for annotation and a description of the annotation process. we also describe which classifiers we use to automatize the classification. before concluding, we discuss our results. ## related work there has been considerable work on the task of predicting personal attributes of users on the basis of their written contributions, e.g. social media posts. personal attributes are typically extralinguistic attributes that are not explicitly mentioned in texts, but can be inferred on the basis of analysis of style, grammar or vocabulary. one can distinguish between physical personal attributes that manifest themselves physically (such as age, ethnicity, gender, etc) on the one hand, and psychological personal attributes including mood, emotion, stress level, sentiment, resilience etc. the independence level that we consider in our work does not fully belong to one of this categories. while independence level is not strictly speaking a physically manifested attribute such as age or gender, it is related to the (observable) behaviour of a person and has thus an obvservable manifestation. yet, it has similar characteristics to a psychological variable in the sense that it is not objectively measurable and is subject to interpretation. an older study was carried out by @xcite in the field of child care to manually analyze written documentations in combination with the environment of children to assess child care behaviour as a basis to predict their preschool behaviour patterns. among the different attributes considered, level of independence is considered. another group where the level of independence is of relevance are old people @xcite . karakaya et al. collected data for 33 elderly people living in a nursing home and 25 elderly living at home. they measured the functional mobility, depressive symptoms, level of independence, and quality of life for both groups for comparison. the level of independence was evaluated by the kahoku aging longitudinal study scale (kals), which evaluates activities of elderly people in 12 areas. each activity was rated on a 4-point scale (0: dependent, 1-2: some help, 3: independent). in this scale, higher scores indicate higher level of inde-pendence. our assessment of independence level follows in essence the proposal of the 4-point scale of karakaya et al. while there has been some work on (manual) independence level classification (e.g. the above mentioned works), we are the first to consider the automatic prediction using learned classifiers on textual documentations in the domain of disability care in social work. furthermore, we investigate how the classification results vary depending on the metzler category considered. an important difference to related work is that level of independence is not predicted from the self-reported experience of subjects, but from the written observations of a third party, in our case the social worker. due to the lack of work on automatic prediction of independence level, we discuss work on the related task of classifying personal attributes. there has been a lot of work on detecting personal attributes, in particular from social media posts from facebook or twitter @xcite . @xcite use logistic and linear regression classifiers to detect psychological attributes related to personality, intelligence, openness and happiness, as well as physical attributes related to sexual orientation, ethnicity, religious and political views, use of addictive substances, parental separation, age, and gender, reaching accuracy levels between 85% and 88% on the data of 58,000 volunteers on facebook. yo and sasahara (2017) tackle the prediction of the personal attributes gender, occupation, and age groups in tweets, reaching accuracy levels of 60-70%. a related task is the detection of stress levels, on which @xcite reach 83%-93% f-measure on four different datasets derived from 350 million tweets data. galatzer-levy et al. (2018) extensively discuss the relevance of different machine learning algorithms and machine learning principles to the study of stress pathology, recovery, and resilience. a related task is the classification of the emotion of a text or social media post. bostan and klinger (2018) compare models for prediction on annotated emotion corpora systematically, performing cross-corpus experiments by training classifiers on each dataset and evaluating them on others, reaching f-measures (micro averaged) of 56% when training on all but one corpus and testing on a held-out corpus and 98% when training and testing on the same corpus. they predict the emotions: joy, anger, sadness, disgust, fear, trust, surprise, love, confusion, anticipation and no emotion (noemo). @xcite constructed the ammer corpus, triggering emotions such as fear, anger, annoyance, insecurity, joy and no emotion in the experimental setting of a car driving situation. they make use of off-the-shelf algorithms and a bidirectional lstm, reaching a f-measure of 76% with transfer learning. @xcite have re-annotated the semeval 2016 stance data set @xcite with the emotion labels anger, anticipation, disgust, fear, joy, sadness, surprise and trust. they apply maximum entropy, support vector machines, a lstm, a bidirectional lstm, and a convolutional neural network (cnn) to provide baseline results for their corpus. they reach best results of 77% f-measure with a bidirectional lstm. ## data the data was collected from 2 different stationary disability care facilities of a major european welfare provider located in the german state of north rhine-westphalia in the context of a common project with bielefeld university. following contractual obligations, the institution will not be named in this paper. the dataset was collected for clients that gave their explicit consent, yielding the complete documentation for 22 clients. the data comprises of 731,601 records with 295,812 observations documented by the social workers in natural language. for each documentation, a client id, a date, and a category is recorded. an example for a documentation is given in the 34 categories describe measures of care applied and originate from the metzler taxonomy that is standard in social care contexts for the purpose of documentation. the seven top level categories of the taxonomy and the respective number of subcategories are given in ## method in this section we discuss the development of guidelines for annotation and provide a description of the annotation process. we also describe which classifiers we use to automatize the classification. ## results we adopt a 5-fold cross-validation setting in which in each fold we train with 80% of the data and evaluate on 20% of held-out data. we report the median f-measure over the five folds for each classifier. as a baseline, we rely on a model trained with unigrams weighted with tf.idf. we refer to this baseline as bag-of-words (bow) model. the only model with a different baseline is the cnn, where the convolutional window is of size 3, corresponding to the trigram features that we use on top of the bow model. therefore, we use the model with word embeddings that are trained on the raw text -rather than stemmed words or lemmata -as a baseline. the baseline score for the cnn is 6% f-measure (macro) (median over five folds). the scores for the different classifiers using the bow model are given in we report the f-measure (macro) for the five best models in first of all, we observe that the macro-average f 1 values in both settings are above the bow baseline by 7% (category-specific setting) and 9% (category-agnostic setting) for the case of linear svm and 7% and 10% for perceptron, respectively. this trend is observed also for logistic regression but not for knn, where the value in the category-specific-setting increases 8% and drops 4% in the category-agnostic setting. for the cnn the value increases 82% in the category-specific setting and 52% in the category-agnostic setting. in general, we observe that the category-agnostic setting outperforms the category-specific setting for most classifiers , i.e. 2% for linear svm, 3% for perceptron, and 1% for logistic regression. the exception is again the knn classifier, where the value is 12% lower. also, the cnn performs 30% better in the category-specific setting. considering the best-performing classifier, i.e. linear svm (one-vs-one) for the specific categories, we observe that for 11 out of 34 categories, the category-agnostic setting performs better than the category-specific setting, for 9 categories both settings perform equally well and for 14 categories the category-specific setting performs better. the category-agnostic setting sometimes outperforms the category-specific setting in cases where there are a low number of (positive) examples for the corresponding category. this is the case for the categories m-21, m-27 and m-28. in m-28, it is not possible to conduct 5-fold cross-validation ## conclusion we have presented an approach to automatically classify observations in written reports by social workers into discrete independence levels reflecting the level of independence of their clients. we have described the construction of a corpus with substantial agreement on the task, comprising of over 15,000 documents. we have presented results on the task with different classifiers, showing that we can obtain f-measures of 95% macro-averaged over the different metzler categories that we considered. we have also shown that a categoryagnostic model outperforms a category-specific approach with one model per category on average over all categories. furthermore, for some categories featuring smaller amounts of examples, the category-agnostic classifier performs better. the cnn model did achieve good results on the task but did not outperform svm with one vs. one. while we did not investigate how to integrate more complex syntactic features other than pos information, this is an obvious avenue for future work. a further interesting question is which other psychological attributes can be predicted on the basis of the documentations considered here. an obvious category to explore is the mood of clients. in addition to classifying independence levels, we intend to perform further studies on the annotated data: one task is to present the trajectories and derived information for independence level per metzler category to social workers to understand if it provides useful insights for them. furthermore, it would be interesting to investigate in how far the observations in written reports by social workers reflect the actual condition or opinion of a client.