Intelligent Analyses on Storytelling for Impact Measurement (2020)

Abstract: This paper explores how Dutch diary fragments, written by family coaches in the social sector, can be analysed automatically using machine learning techniques to quantitatively measure the impact of social coaching. The focus lays on two tasks: determining which sentiment a fragment contains (sentiment analysis) and investigating which fundamental social rights (education, employment, legal aid, etc.) are addressed in the fragment. To train and test the new algorithms, a dataset consisting of 1715 Dutch diary fragments is used. These fragments are manually labelled on sentiment and on the applicable fundamental social rights. The sentiment analysis models were trained to classify the fragments into three classes: negative, neutral or positive. Fine-tuning the Dutch pre-trained Bidirectional Encoder Representations from Transformers (BERTje) (de Vries et al., 2019) language model surpassed the more classic algorithms by correctly classifying 79.6% of the fragments on the sentiment analysis, which is considered as a good result. This technique also achieved the best results in the identification of the fundamental rights, where for every fragment the three most likely fundamental rights were given as output. In this way, 93% of the present fundamental rights were correctly recognised. To our knowledge, we are the first to try to extract social rights from written text with the help of Natural Language Processing techniques.

## introduction in leuven, belgium, there are many charitable organisations that support socially vulnerable people. for evaluating the progress of their work, each of them has their own system, mostly handwritten on paper. in 2018, the local community organisation vzw buurtwerk 't lampeke started a cooperation with software company kunlabora. they wanted to obtain qualitative insights in their coaching, since they were lumbered with a lot of administration. the result was a tailor-made software tool named mezuri foot_0 , a java application for organisations supporting socially vulnerable people to understand and measure the impact of their coaching tracks. in this tool, collaborators called bridging coaches can keep diary fragments (written in dutch) for different families. in this way, the bridging coaches can, with the help of intelligent analyses, keep track of how the family is doing and which fundamental social rights (regarding education, work, etc.) are acquired. the most important aspect of mezuri is that these diaries have open-ended instead of closed-ended inputs. this allows the coaches of the organisations to write free text and focus on the family instead of having to tick boxes or fill in scales. it is then the task of the mezuri programme itself to get more objective, scale-like information out of these text fragments with the help of intelligent analyses. in this paper, there is a focus on improving the following two algorithms important for the bridging coaches: this paper investigates which algorithms obtain the most accurate analysis on these dutch text fragments. this involves several challenges. firstly, little data exists and the available data are private. secondly, the diary fragments consist of subjective information which is also not always 100% correct: sometimes, a family does not immediately tell the truth or perhaps glosses over reality. this make it even more difficult to objectify and quantify the information written in the diary fragments. contributions 1) we show that by fine-tuning the existing bertje language model on classifying a dutch diary fragment into three classes (negative, neutral, positive) an accuracy of 80% can be reached. 2) we show that this technique can also be used to fine-tune the model to recognise fundamental rights: when for every fragment the three most likely rights are given as output, 93% of the present rights are correctly recognised. ## background and related work sentiment analysis (sa) is a hot topic in natural language processing (nlp). it is often used on reviews or social media posts to monitor the reputation of a service, person or product. a text fragment is then classified as positive or negative (i.e. binary classification) or, in case of a ternary classification, as neutral. in the past, often lexicon-based @xcite or machine learning with bagof-words @xcite approaches were used for sentiment analysis. more recently, the use of embeddings @xcite and neural networks became more popular (prabha and umarani srikanth, 2019) in nlp. however, a disadvantage of using neural networks is the large amount of training data they require, which can be limited by using transfer learning. in nlp, this is often done using pre-trained language models. bert (bidirectional encoder representations from transformers) is such a language model made available by google @xcite . bert has proven to achieve state-of-the-art results on various tasks including sentiment analysis, as in the study of @xcite where english 1-sentence movie reviews were classified into 5 classes, reaching accuracies of up to 84%. therefore, it is investigated whether this approach can also achieve high performance on sentiment analysis with the mezuri dataset. however, the dataset of this paper contains dutch text fragments and mostly longer than one sentence, making the task more complex. since december 2019, there also exists a dutch bert model @xcite . trained on 2.4 billion dutch tokens, this monolingual model outperforms bert's equally-sized multilingual model in various tasks, including sentiment analysis. to this end, bertje will be used instead of the multi-language version of bert. there is a lot of research done on sa in other use-cases. for example, @xcite classified customer reviews of hotels as good or bad (i.e. binary classification) with a lexicon-based approach yielding an accuracy of 90%. @xcite uses machine learning algorithms to classify tweets into 3 different classes achieving an accuracy equal to 70%. however, the task in mezuri is more difficult than the task in @xcite in several ways. firstly, in mezuri, a fragment is classified into three classes instead of two. secondly, the fragments in mezuri are written in dutch, a language on which less research has been done than on english. lastly, assigning labels to the fragments of mezuri is a subjective task, while when two people label reviews or social media posts, they will probably reach a higher agreement score. to our knowledge, we are the first to try to extract social rights from written text. this is considered a multi-label problem, as a single fragment can contain multiple social rights. the task is then to predict the set of correct labels. this is different from the sentiment analysis task, where a fragment belongs to a single class. according to @xcite , there are three ways to tackle the multi-label classification problem: adapt the method, transform the problem and ensembles. as described by @xcite , the first method is based on the idea to adapt the single-label methods in a way they can cope with multi-labelled data. a method that uses this principal is multi-label k nearest neighbours (mlknn) @xcite . an advantage of this method is that the correlations between the labels are taken into account. the second idea is to transform the multi-label problem into multiple single-label problems. binary relevance, classifier chains and label powerset @xcite use this approach. a third manner of extracting social rights in a supervised way is with ensemble methods. an example is rakel @xcite , where random k-labelsets are given to the label powerset method. ## dataset about ten bridging coaches of caw oost-brabant and werfgezinscoach wrote the diary fragments in which they reflect on a meeting with a family. together they coached nineteen families, from which they made 460 high-quality, dutch text fragments available. for this project, they anonimysed these text fragments by replacing the names with initials. the original diary fragments have an average of about 188 words per diary fragment with a standard deviation of 198. ## methods before feeding the data to the machine learning model, the data needs to be standardised. this can be done by removing the special characters and the unnecessary blank spaces. furthermore, the replacement of capital letters by lowercase letters, the removal of stop words and stemming or lemmatisation @xcite is investigated. to avoid bias (i.e. assigning a sentiment to gender specific pronouns), personal pronouns and names figure 4: distribution of the number of social rights in a fragment (one fragment can contain several social rights) (i.e. initials) of persons are replaced by the words persoon (person) and naam (name) respectively. next, the text fragments need to be vectorised (i.e. transformed into a numerical representation). in this paper, this is performed in two ways. firstly, with a bag-of-words (bow) approach @xcite and secondly, with word embeddings @xcite . ## conclusion in this paper, we investigate the best way to perform sentiment analysis and extract social rights from subjective dutch text fragments with the help of manually given labels. the results demonstrate that fine-tuning bertje outperforms other techniques with an accuracy of 80% on sentiment analysis and 93% on extracting social rights when using the 3 most probable rights as output. further research directions could explore other pre-trained language models or exploit automatic data augmentation.