Fora: A corpus and framework for the study of facilitated dialogue (2024)

Abstract: Facilitated dialogue is increasingly popular as a method of civic engagement and as a method for gathering social insight, but resources for its study are scant. We present Fora, a unique collection of annotated facilitated dialogues. We compile 262 facilitated conversations that were hosted with partner organizations seeking to engage their members and surface insights regarding issues like education, elections, and public health, primarily through the sharing of personal experience. Alongside this corpus of 39,911 speaker turns, we present a framework for the analysis of facilitated dialogue. We taxonomize key personal sharing behaviors and facilitation strategies in the corpus, annotate a 25% sample (10,000+ speaker turns) of the data accordingly, and evaluate and establish baselines on a number of tasks essential to the identification of these phenomena in dialogue. We describe the data, and relate facilitator behavior to turn-taking and participant sharing. We outline how this research can inform future work in understanding and improving facilitated dialogue, parsing spoken conversation, and improving the behavior of dialogue agents.

## introduction in light of rampant toxic polarization on social media @xcite , sliding trust in democratic institutions, and the shortcomings of traditional methods of seeking and using public opinion, new methods for responsibly seeking participatory insight are critically needed across many fields, including for civic use in governance and for nuanced insight in the social sciences. social media has been used as a source of insight into public experience and discourse by the natural language processing (nlp) and computational social science communities for decades now, but welldocumented user self-selection and polarization make trusting insights drawn from social media difficult. while surveys and other standard social research methods have strong empirical grounding, they often limit opportunities for free-form comment by participants. facilitated dialogue provides one alternative avenue of seeking input and insight from various stakeholders, community members, constituents, or other participants. one example use case for facilitated dialogue is for community listening. members of the community can be invited to participate in small group conversations, where they are prompted by a trained facilitator to share their personal experiences and perspectives regarding a specific topic that is important to the community. analysis of patterns across conversations can serve as useful input to decision-making processes that impact the community, such as public policy formation or a hiring process for a public servant. participation in such community engagement processes may also strengthen "civic muscle." in this work, we present fora, 1 a corpus of annotated transcripts of 262 multi-person facilitated dialogues similar to the one outlined above. these conversations were organized with various usbased organizations that partnered with the nonprofit cortico. their goal was to engage members or stakeholders by inviting them to converse in an effort to understand their personal experiences and perspectives on topics like education, public health, and upcoming elections. along with this corpus of facilitated dialogue, we present a general framework for the analysis of facilitated group dialogue through outcome metrics relevant to evaluating a facilitation process, including turn-taking. we taxonomize key sharing behaviors and facilitation strategies. we run a human annotation effort for a 25% (10,000+ speaker turns) subset of the corpus. we use it to train or prompt and evaluate the performance of several language models automating the identification of facilitation strategies and sharing behaviors, thus establishing baselines on tasks essential to the study of facilitated dialogue. moreover, we describe the data in terms of facilitator behavior and relate facilitator behavior to turn-taking and participant sharing. we outline how this research can inform work to understand and improve facilitated dialogue for facilitators and participants, and we discuss its usefulness in future work on parsing spoken conversations, and improving dialogue agents. ## related work the study of conversation spans many disciplines including linguistics, nlp, and the social sciences. building on speech act theory, major efforts to taxonomize, annotate, classify, and model dialogue acts have been important to the foundation of nlp. renowned contributions include the switchboard corpus @xcite taxonomy of speech acts, damsl corpus @xcite , early work from @xcite on tagging speech acts in dialogue, and dialogbank, an interoperable system of dialogue act annotation @xcite . in parallel, scholars in other fields have noted the importance of dialogue to public understanding, especially in the face of social conflict. @xcite lay the groundwork for academic work on understanding both the value of facilitated dialogue for collective understanding, as well as the unique role of the facilitator in setting the norms and guiding the conversation towards emergent expression. the field of deliberative polling makes a case for the importance of deliberative discourse in leading to better decision outcomes @xcite and the legitimization of democratic outcomes @xcite . @xcite discusses the importance of sustained dialogue for changing human relationships in the face of deeply-rooted conflict, and @xcite notes the importance of storytelling in productive deliberation forums. others note the power of narrative for digital participation @xcite and democracy @xcite . corpora of public dialogue outside the social media space are rare, and high-quality annotated spoken conversation transcripts are even rarer. word distribution and turn-taking patterns in speech and transcribed speech are distinct from written text, making spoken speech datasets for understanding community insight important. the recent launch of the candor corpus @xcite ) marks an important leap in conversation research. eu-ropolis @xcite , a set of discussions from a transnational european deliberative poll, provides another related dataset, and research from dillard (2013) finds a relationship between facilitation types and the trajectory of public deliberations. dialogue can function as a method, tool, or input to a process of gradual understanding of a problem, including in a research process. many fields of social research use similar methods of surfacing insight in a group setting, including group interviews, discussion groups, and focus groups. focus groups are a common and validated method of surfacing insight from a group @xcite and they have existed for 80 years @xcite . there are documented and commonly used practices for running focus groups @xcite but to our knowledge, there are no known datasets for their study at scale. corpora of recorded meetings like one described in @xcite exist, but facilitated conversations differ from meetings in important ways. a rich theory on facilitation techniques comes from literature in education. research from @xcite shows that facilitative moves can have an impact on the outcome of interest to the facilitation setting, and that "request" utterances by the facilitator encourage response. research from chung (2011) on education discussion fora relates the presence of nine facilitation techniques on those fora, including summarization, making connections, providing opinions, and inviting feedback, to the presence of critical thinking by participants in online discussions. research on facilitation for teaching describes three main functions of facilitation: social, organizational, and intellectual @xcite . the function role of the facilitator, according to @xcite is to manage group discussions and processes so that group members have a positive experience, to host a process that "promotes valuable results in group dialogue, analysis, and planning" and to use techniques that aid in group interaction or the accomplishment of those goals. ## motivation hand-transcribed multi-person conversation datasets of substantial size are relatively rare in the nlp community, and to our knowledge, no corpora of annotated and analyzed semi-structured facilitated dialogues currently exist. furthermore, as online discussion platforms look to promote civil discourse, experiments in using automated agents to facilitate and moderate discussion or conduct interviews are already underway @xcite @xcite . as such, understanding human facilitation is crucial to the successful development of productive conversations, including ones that may involve an automated agent. as facilitated dialogue emerges as a forum for discourse and public input in contrast to discourse online, a better understanding of facilitation behaviors can inform the design of productive group conversations, no matter their intended outcome or purpose. ## corpus the fora corpus is a set of conversation transcripts, conversation guides, and annotations associated with 262 conversations spread across 16 conversation collections convened over a variety of topics in diverse communities. each conversation collection was convened by a partner organization in the united states which, in partnership with cortico, a us-based social technology non-profit, hosted a series of conversations of similar structure about a community issue or series of issues. conversation collections range in size from a minimum of 5 conversations to a maximum of 76 conversations. each conversation collection has a companion conversation guide, developed by the hosting partner organization in partnership with cortico. facilitator roles in the conversation were hand-labeled by our researchers. most conversations are facilitated by one individual. 29 are co-facilitated, of which 28 of which were facilitated by 2 individuals and a single one was co-facilitated by three individuals. conversations in the fora corpus occurred between november 14th, 2019, and april 6th, 2023. conversations were held with a median of 6 people, with a minimum of 2 and maximum of 12. the average conversation length was 66 minutes, with a minimum of 8 minutes and a maximum of 117 minutes, or just under 2 hours. conversations had an average of 152 speaker turns, with a minimum of 14 turns and a maximum of 719. most conversations were held online over meeting software. conversations were recorded with consent and transcribed by rev transcription software, and partners were able to redact and edit transcripts for mistakes following the conversation. a visualization of the duration of speaker turns across participants in a typical conversation in the fora corpus can be seen in figure 1 . the appendix provides more descriptive statistics on the conversation collections within the corpus. ## characterizing facilitated dialogue we first characterize facilitated dialogue using a number of computational metrics to understand variation in the corpus across collections. ## defining participant sharing following the presentation of high-level computational measures of turn-taking in the corpus, we turn our attention to annotating speaker turns in the corpus according to several codebooks which we define. because fora conversation guides were designed to elicit personal stories and experiences from participants, we develop a codebook and annotation task related to identifying two types of sharing, personal story and personal experience, as the main outcomes of interest from the dialogue. personal story: building on antoniak et al. (2023), we define a personal story as a series of events involving the speaker that occurred in the past or in the ongoing active present. personal experience: following a similar approach in @xcite to defining "experiential knowledge" in contrast to story, we define personal experience as a mention of personal background, identity, fragmentary recall of events, or mentions of habitual or ongoing actions, occurrences, or speaker feelings within a speaker turn. we choose speaker turns as our unit of labeling for the annotation task, and use the above definitions to apply labels to all speaker turns from a sample of 70 (26%) conversations from the corpus. to sample a subset of conversations from the corpus for annotation, we first sampled one from each of the 16 collections, then randomly sampled other conversations until we hit 25% of the corpus, or 66 conversations. we annotated 4 co-facilitated conversations as well, bringing the total number of annotated conversations to 70 conversations. we exclude co-facilitated conversations from model training and analysis on the facilitation strategy tasks due to potential differences in co-facilitated conversations that are not the focus of this work. speaker turns could be annotated with both labels if both kinds of personal sharing occurred within a turn. for example, this turn contains both: my name's addison. i'm from machias in washington county. and a time i felt like i mattered to my community was when my family was going through a hard time and some people from around our community sent presents on christmas, and also helped with some of the bills. addison's name and place of origin count as mentions of personal experience, while her story about going through a hard time, being sent presents, and being helped with bills qualifies as a story due to its described sequence of events. ## results for each gpt-4 foot_2 response, we extract the string representing the expected label in order to derive a binary value for the label for that sample. to evaluate performance, we treat the problem as a singleclass classification problem for each task for maximum comparability across conditions. we fine- ## research directions for the study of facilitated dialogue we hope these data, measures, tasks, and analyses can encourage future work in this area, especially combined with behavioral experiments and other methods and insights across fields. we outline some areas of potential work: ## conclusion & future work we present a unique corpus of annotated facilitated conversations and a suggested framework for analyzing facilitated dialogue. we outline key tasks for understanding facilitated dialogue in terms of participant sharing behavior and facilitator behavior, create a 25% sample of human-labeled data, then establish baselines for each task. we find decent model performance on some tasks, and especially for rare phenomena, gpt-4 does decently well on these tasks in contrast to a fine-tuned roberta model. in a data-poor context where fine-tuning is challenging, this points to the potential for large models to assist in making sense of small data, pending validation. future work can investigate performance using open-source models on these tasks. overall, we find that fora conversations regularly elicit personal stories and experience. we find that some facilitation strategies are more common than others, and some strategies may relate to increased rates of participant personal sharing. we believe this corpus, framework, and findings will fuel research on facilitated conversations among humans, effective facilitation of group dialogue, modeling goal-driven dialogue, understanding spoken stories in dialogue, and facilitation of groups of humans by language agents. we hope this work will serve as a starting point for a rich emerging science of facilitated dialogue. future work can compare conversation norms in facilitated dialogue to more traditional forms of interviewing, information collection, and conversation. comparison datasets in the future could include podcast data or tv interviews with multiple conversationalists. in this work, we did not test every combination of prompting methods and training parameters, instead focusing on a high-level framework, definitions, and establishing baselines. there remains significant opportunity for improvement, including exploration of methods to improve finetuning for small classes. future work can examine additional prompting strategies like randomly sampling from the training data to provide few-shot examples, as well as chain-of-thought prompting.