Tailoring Vaccine Messaging with Common-Ground Opinions (2024)

Abstract: One way to personalize chatbot interactions is by establishing common ground with the intended reader. A domain where establishing mutual understanding could be particularly impactful is vaccine concerns and misinformation. Vaccine interventions are forms of messaging which aim to answer concerns expressed about vaccination. Tailoring responses in this domain is difficult, since opinions often have seemingly little ideological overlap. We define the task of tailoring vaccine interventions to a Common-Ground Opinion (CGO). Tailoring responses to a CGO involves meaningfully improving the answer by relating it to an opinion or belief the reader holds. In this paper we introduce Tailor-CGO, a dataset for evaluating how well responses are tailored to provided CGOs. We benchmark several major LLMs on this task; finding GPT-4-Turbo performs significantly better than others. We also build automatic evaluation metrics, including an efficient and accurate BERT model that outperforms finetuned LLMs, investigate how to successfully tailor vaccine messaging to CGOs, and provide actionable recommendations from this investigation.Tailor-CGO dataset and code available at: https://github.com/rickardstureborg/tailor-cgo

## introduction finding common-ground has long been understood as an important aspect of language and communication @xcite . creating this connection during communication can establish a sense of trust and is an inclusive goal to strive for when preparing personalized messaging. we can establish common ground on many things, such as beliefs or opinions, although it may be unclear whether such opinions can be seamlessly related to the main message being communicated. one particularly relevant use case for establishing common ground is vaccine messaging. loss 1 tailor-cgo dataset and code available at: https: //github.com/rickardstureborg/tailor-cgo {concern} you can get a fever from the vaccine. the military acts in our best interest. your immune system is like your body's military force; prepared for any invaders that threaten our well-being. vaccines are like a training exercise: presented with a harmless piece of the invader (like a protein from the virus). it's similar to how the military practices with drills and simulations, which might be tough, but help ensure a strong defense. most side effects such as a fever or tiredness are signs this training is working. side effects are insignificant compared to the actual threats out there. ## tailor-cgo dataset creation in this section we describe the task of tailoring a response to a cgo, introduce and describe the dataset and its components, and outline how the dataset is labeled. the final dataset contains 22, 400 unique tailored responses from 6 different llms, labeled with a mix of absolute scores or pairwise rankings. ## automatic evaluation while human annotation shows a high agreement on evaluating llm responses, conducting it at scale to study the various settings we consider is prohibitively expensive. hence, in this section we use the annotations collected above to develop automatic evaluation metrics for the tailor-cgo task. the goal of automatic evaluation is to provide a cheap alternative for labeling candidate responses and automatically evaluate or compare models. it also allows for deeper analysis of trends when human annotation becomes restricted by scale. we describe our approach below. first, we prompt a generative model (gpt-4-turbo) to score responses directly. second, to further reduce cost, we fine-tune open-source language models (bert and llama-2) using the results from the former and/or human annotations. ## results to better explore the usefulness of tailor-cgo towards improving our understanding of tailored responses, we investigate what models, strategies, and opinions work best. this section describes a series of analyses looking at factors to consider when generating responses tailored to cgos, which may be of interest to nlp researchers as well as public health professionals working on vaccine hesistancy. lacking quantity poor quality fallible science imperfect protection herd immunity natural immunity insufficient risk existing alternatives direct transmission harmful ingredients specific side effects dangerous delivery high-risk individuals religious and ethical beliefs right to autonomy incompetence profit motives censorship conspiracy community health crime/security job/career personal finance healthcare system future corporations, banks, tech & automation race personal health gender & sexuality self-perception and values immigration status in life global attitudes and foreign policy political issues education discrimination relationships and family economy and inequality news, social media, data, privacy science leadership religion 1.0 1.5 2.0 2.5 3.0 3.5 4.0 brighter colors indicate higher scores, while white squares are nulls that were not sampled during annotation. religion, while an opinion topic that scores poorly in our testing, seems to provide useful opinions for tailoring when focusing on the direct transmission concern (see appendix f.3 for an example output). ## conclusion we introduce tailor-cgo, a comprehensive and high-quality dataset for training and evaluation of tailoring vaccine interventions to common-ground opinions. we benchmark several major llms, finding that gpt-4-turbo best tailors responses to cgos. we build evaluation metrics on top of this dataset to allow cheap and accurate evaluation of models. finally, we analyze which opinions are better suited to tailor vaccine interventions with, and provide recommendations for which opinions to select for specific concern categories.