Are Experts Needed? On Human Evaluation of Counselling Reflection Generation (2023)

Abstract: Reflection is a crucial counselling skill where the therapist conveys to the client their interpretation of what the client said. Language models have recently been used to generate reflections automatically, but human evaluation is challenging, particularly due to the cost of hiring experts. Laypeople-based evaluation is less expensive and easier to scale, but its quality is unknown for reflections. Therefore, we explore whether laypeople can be an alternative to experts in evaluating a fundamental quality aspect: coherence and context-consistency. We do so by asking a group of laypeople and a group of experts to annotate both synthetic reflections and human reflections from actual therapists. We find that both laypeople and experts are reliable annotators and that they have moderate-to-strong inter-group correlation, which shows that laypeople can be trusted for such evaluations. We also discover that GPT-3 mostly produces coherent and consistent reflections, and we explore changes in evaluation results when the source of synthetic reflections changes to GPT-3 from the less powerful GPT-2.

## introduction motivational interviewing @xcite is a highly effective counselling practice in healthcare @xcite , where the therapist focuses on evoking the client's own motivation for behaviour change, such as smoking cessation and alcohol use reduction. in mi, reflective listening is a crucial strategy of showing empathy, where the therapist conveys a brief conversational summary of how they understand what the client said @xcite . an example is shown in learning effective reflective listening requires considerable training time and expert supervision @xcite . therefore, recent studies used language models (lms) as automatic reflection generators to aid training @xcite ; ahmed, context client: well, i'm here because my mom wants me to be here. therapist: mm-hmm. client: i don't really wanna be here, but it-it-whatever. therapist: got it. client: um, she-she found my stash-therapist: uh-huh. client: -and she freaked out, and she's going crazy over it. um, i don't why she was going through stuff in the first place, but whatever, so, now i'm here. ## conclusion in this work, we probed whether laypeople can be used as an alternative to experts in evaluating coherence and context-consistency of counselling reflection generation. accordingly, we asked both laypeople and experts to annotate synthetic reflections generated by lms and human reflections from actual therapists. we found that both laypeople and experts are reliable annotators and that they also show moderate to strong inter-group correlation, which is the first concrete evidence that laypeople are capable of such annotations, although laypeople are relatively less aligned with experts on gpt-3 reflections. furthermore, we found that gpt-3 is mostly able to generate coherent and consistent reflections, and we also explored the annotation shift on human reflections when the source of synthetic reflections changes from the smaller gpt-2 to the more powerful gpt-3. for future work, we plan to mix, in each batch, synthetic reflections from models of different scales, and investigate how the resulting human evaluations might differ. another direction worth exploring is alternative ways of coherence annotation, such as ranking, for more nuanced human evaluation results. future work may also re-examine and modify the error categories to increase iaa on error annotations. we also leave potentially iaaimproving annotation procedures to future work, such as using a warm-up exercise task before actual annotation and allowing annotators to discuss with each other to resolve their differences.