Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations (2022)

Abstract: Reflection is an essential counselling strategy, where the therapist listens actively and responds with their own interpretation of the client’s words. Recent work leveraged pre-trained language models (PLMs) to approach reflection generation as a promising tool to aid counsellor training. However, those studies used limited dialogue context for modelling and simplistic error analysis for human evaluation. In this work, we take the first step towards addressing those limitations. First, we fine-tune PLMs on longer dialogue contexts for reflection generation. Then, we collect free-text error descriptions from non-experts about generated reflections, identify common patterns among them, and accordingly establish discrete error categories using thematic analysis. Based on this scheme, we plan for future work a mass non-expert error annotation phase for generated reflections followed by an expert-based validation phase, namely “whether a coherent and consistent response is a good reflection”.

## introduction patient health can be greatly improved by changing behaviours such as smoking and alcohol consumption. as patients rarely ask for help with it, healthcare practitioners often need to encourage, counsel and advise them to make changes @xcite . an effective counselling approach for this purpose is motivational interviewing @xcite , which aims to elicit the motivation for change from the client 1 themselves. in particular, reflection -also known as reflective listening -is an essential conversational strategy in mi that has been shown to be related to positive counselling outcomes @xcite . a good reflection conveys to the client that the therapist is listening, hearing and understanding them 1 a person receiving mi is not necessarily a patient, therefore we use "client" instead of "patient" in this work. ## conclusion in this work, we explored non-expert annotation of machine-generated reflections for counselling dialogues, based on the assumption that non-experts are capable of context-informed 1) judgement of whether a reflection is coherent and consistent and 2) identification of the errors in an incoherent/inconsistent reflection. we identified common patterns among the free-text error descriptions from non-experts about generated reflections and accordingly used thematic analysis to establish discrete error categories that emphasised context understanding. based on these categories, we plan for future work a) a mass non-expert error annotation phase for generated reflections, followed by b) an expertbased validation phase, and the results from both phases will be released to the public.