Soft Measures for Extracting Causal Collective Intelligence (2024)

Abstract: Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.

## introduction social science has long sought to understand and model the collective intelligence underlying humanity's most pressing problems such as climate change, sustainable food supply, and violent conflict driven by inequitable resource distribution. these are social-ecological systems (ses) problems characterized by complex, interwoven feedback loops involving human and natural systems @xcite . to model collective intelligence about ses, we can leverage mental models of causal system structure. researchers in the social sciences have formally encoded ses mental models using fuzzy cognitive maps (fcms) that represent causal systems as signed, weighted digraphs, where edges represent causal relationships among natural language concepts @xcite like that depicted in figure 1 . fcms are inspired by human causal mental models that people use to explain causal mechanisms and generate predictions @xcite . fcms have been widely used to facilitate crossdisciplinary communication within research teams @xcite , make qualitative and numerical predictions, and assess collective intelligence @xcite @xcite . however, little work has explored extracting high-integrity fcms from textual corpora. the development of a robust text-to-fcm method would enable more rapid synthesis of science-and stakeholder-informed perspectives to provide access to latent collective intelligence about ses. this work presents a natural language processing (nlp) approach to (1) extracting fcms from text with large language models (llms) and (2) measuring extracted fcm quality with novel soft f1 measures that permit approximate semantic matches rather than requiring exact node and edge matches. this helps capture and accumulate diverse causal collective intelligence of ses domains. we provide code and datasets for reproducibility. 1 2 background and objectives extracting fcm edges is a specialization of extracting semantic relations or causal graphs: each node is a textual span describing causal factors, and each edge is a directed causal increase or decrease relationship where the source of the edge quantitatively increases or decreases the target. two issues distinguish fcm extraction from previous graph-based extraction tasks: (1) high expressiveness and (2) partial correctness. we describe these two issues briefly to motivate our approach. high expressiveness. causal variables (i.e., fcm nodes) and relations may be expressed multiple ways, so human annotators may disagree and a machine prediction may actually outperform a pre-determined human-generated gold standard according to human judges. many graph extraction approaches are validated by their proximity to a singular gold standard generated by human experts, where small deviations from the standard greatly affect the measure. this includes neural network loss functions. some relaxed matching strategies help account for textual containment or overlap @xcite , but these have not been adequately extended to graph extraction. partial correctness. a sub-optimal node or edge that captures a valid causal relationship is often useful to report as a component of an fcm-especially in data-poor contexts, as even limited information can improve the understanding of a given systemso an nlp model's capability to produce partially correct edges is important to capture ( source target direction issue turbine structures blue mussels increase na (gold standard) numbers of blue mussels turbine structures increase source & target swapped; extra source text turbines mussel populations decrease simplified source; extra target text; incorrect direction turbine structures blue mussels decrease incorrect direction ## discussion and conclusions this paper presents an evaluation of fine-tuned llm-based methods for extracting fcms from text, while also introducing and assessing novel edge-based similarity measures to validate the quality of these fcms. the study highlights the limitations of traditional measures, which often fail to capture the nuances and partial correctness in fcms, especially in the context of ses research. for instance, there were cases where llms outperformed human annotators, yet these instances might have been overlooked if we relied solely on traditional measures like the f1 score or validation set loss. the novel edge-based measures allowing for partial positives show markedly greater correla-tion with human judgments. in addition to examining similarity measures, this work initiates a new approach for validating graph-based nlp predictions by employing pairwise comparative human judgments, using the elo rating system, to rank predictions. this method was used to compare human-derived rankings with those generated by similarity measures, providing a proof of concept for enhancing the qualitative evaluation of nlp-generated fcms. fine-tuning llms proved beneficial, resulting in higher bleu-e scores and improved model performance. whereas the validation set losses for all fine-tuned llms appear similar (figure 3 in appendix c), their qualitative performances differed significantly. although bleu-e offers a more accurate assessment than validation set loss, these improvements do not fully resolve the underlying issues with the current similarity measures. the study's findings emphasize the necessity of developing more specialized measures that are better aligned with human judgment and capable of capturing the complexities of fcm extraction. this study represents an initial step towards that goal, highlighting the limitations of current approaches and setting the stage for future research. future work will focus on developing and validating new similarity measures that can better capture the complexities and partial correctness in fcm extraction. for instance, greater correlation with human judgment should be achievable by parameterizing tp and pp with scalars and employing optimization. additionally, integrating human-inthe-loop approaches may help refine llm outputs, leading to more accurate fcms. we also imagine a range of applications and extensions. for instance, the proposed measures can be straightforwardly extended to knowledge hypergraphs. furthermore, by symmetrizing any one of our typically asymmetric measures we may interpret it as a kernel @xcite . explicitly, let @xmath0, and define a kernel k as, this interpretation brings to bear the entire suite of kernel methods for the study of fcms to facilitate visualization, classification, and general pattern recognition. in conclusion, this study has provided insights into the evaluation of llm-generated fcms and also underscores the need for continued research. our framework provides a structured approach for these evaluations. this paper marks just the beginning of a journey towards improving the overall evaluation framework for fcms and enhancing the role of llms in collective intelligence research, particularly in ses contexts with small quantities of low quality textual data.