An LLM Feature-based Framework for Dialogue Constructiveness Assessment (2024)

Abstract: Research on dialogue constructiveness assessment focuses on (i) analysing conversational factors that influence individuals to take specific actions, win debates, change their perspectives or broaden their open-mindedness and (ii) predicting constructiveness outcomes following dialogues for such use cases. These objectives can be achieved by training either interpretable feature-based models (which often involve costly human annotations) or neural models such as pre-trained language models (which have empirically shown higher task accuracy but lack interpretability). In this paper we propose an LLM feature-based framework for dialogue constructiveness assessment that combines the strengths of feature-based and neural approaches, while mitigating their downsides. The framework first defines a set of dataset-independent and interpretable linguistic features, which can be extracted by both prompting an LLM and simple heuristics. Such features are then used to train LLM feature-based models. We apply this framework to three datasets of dialogue constructiveness and find that our LLM feature-based models outperform or performs at least as well as standard feature-based models and neural models. We also find that the LLM feature-based model learns more robust prediction rules instead of relying on superficial shortcuts, which often trouble neural models.

## introduction numerous prior studies analyse or forecast the constructiveness of conversations spanning a wide range of topics. these include the examination of dialogues aimed at persuading individuals to take specific actions like contributing to charitable causes @xcite , shifting their stance @xcite figure 1 : flowchart delineating a high-level overview of our proposed framework for dialogue constructiveness assessment. 2020), garnering more votes @xcite , argumentation to achieve consensus @xcite @xcite , and investigating how dialogues foster openminded thinking @xcite . two main goals of these studies are to (i) better understand the interplay between linguistic patterns humans exhibit and the targeted outcomes associated with dialogue constructiveness and (ii) build models that can anticipate the outcome of constructive dialogues for certain use cases. these goals are usually achieved by building two types of models: feature-based models, which take linguistic features as input to predict the targeted outcome, and neural models like pre-trained language models (plm), which take adialogue (or a part of it) as input. the former is usually an interpretable model (e.g., logistic regression), while the latter can be artificial neural networks like bert @xcite , longformer @xcite and few-shot decoder-only transformers @xcite that tend to produce higher task accuracy. however, these two types of models have downsides. for example, the input data of feature-based models often require expensive human annotations (e.g., levels of disagreements and coordination tactics) @xcite @xcite , whilst neural models are black-boxes. further, these models are susceptible to learning shortcuts-decision rules that perform well for the majority of benchmark examples but do not hold in general @xcite . for instance, neural models are prone to learning shortcuts 2 due to the exposure to a rich input space. likewise, feature-based models can learn shortcuts, though this issue can be mitigated by careful feature engineering. to our knowledge, prior works use distinct sets of features without a unifying framework, many of which were tailored to a specific context (e.g., specific persuasion tactics for soliciting donations). this work contributes by introducing a unifying framework for dialogue constructiveness assessment (figure 1 ) that capitalises on the strengths of both model types while mitigaiting their shortcomings. it integrates both simple algorithmic rules and prompting a large language model (llm), namely gpt-4 @xcite , to automatically extract a rich set of dataset-independent 3 linguistic features, such as the average occurrence and usage change of collaboration markers @xcite and dispute tactics @xcite used throughout a given dialogue. we will refer to the features generated by heuristics and llms as discrete and llmgenerated features, respectively. these features are 2 we use the terminology introduced by @xcite . other works also refer to shortcuts as "biases" or "spurious correlations". 3 namely, these features that our framework provides are applicable to any dataset related to dialogue constructiveness and potentially adaptable to other dialogue-style datasets. ## llm feature-based framework the llm feature-based framework that we introduce works as follows: given a labelled dataset of dialogue constructiveness, we generate linguistic features at (i) the utterance-level for all utterances or (ii) at the dialogue-level, for every input dialogue. more concretely, we include six linguistic feature sets ( once the linguistic features are generated and validated, we compute statistics for each aforementioned feature annotated at the utterance-level (i.e., all features sets except for qoa) to capture the dynamics of dialogue; the resultant statistics would the politeness strategies introduced by @xcite and implemented in convokit @xcite , encompassing greetings, apologies, directness levels, and the use of polite expressions. collaboration markers conversation markers that are indicative of collaborative discussions @xcite , including the introduction and adoption of ideas, expressions of uncertainty or confidence, pronoun usage patterns, and linguistic style accommodation. dispute tactics indicators concerning the use of distinct rebuttal strategies (i.e., attempts to counter the arguments of an opponent) and coordination tactics (i.e., attempts to promote understanding and consensus), introduced in @xcite . quality of arguments (qoa) assessing the qoa has been an important area of research in linguistics and nlp @xcite . despite the subjectivity, it can be generally agreed that arguments based on scientific evidence are more convincing than those based solely on speculation from an individual without expertise on the subject matter. we explore a new way to model argument quality: prompting an llm to grade the quality of all arguments used in a discussion on a 0-10 scale. ## modelling and evaluation we train all models (except for n-shot gpt-4o baselines) with seven-fold flat cross-validation @xcite . this method can provide a reliable assessment of model performance while drastically lowering the computational load. it is especially useful for moderately sized datasets like ours and can produce results similar to nested cross-validation for models with relatively few hyperparameters. appendix a.6 details the hyperparameter-tuning, training budget and computing infrastructure. for the evaluation of classifiers, we use the area under the receiver operating characteristic curve (auroc) and the area under the precision-recall curve (aupr). for regressors, we use spearman's rank correlation and mean absolute error (mae). we report the average and standard deviation of the results from three seeds for all models except for the 20-shot prompted gpt-4o baseline to due to its cost. 12 feature-based models were trained using scikit-learn library @xcite , while neural models were trained on the transformers library @xcite using the pytorch backend running on python 3.11. foot_8 6 experimental results ## conclusions in research on dialogue constructiveness, the seemingly inevitable trade-off between cost, interpretability, and model performance has long been a problem. to break this curse, we propose a framework that provides a battery of dataset-independent linguistic features and can be leveraged to build llm feature-based models for dialogue constructiveness assessment at a reasonably low cost. in modelling dialogue constructiveness across three datasets-opening-up minds, wikitactics, and articles for deletion corpus-our results show convincing evidence that the llm feature-based models consistently outperform or performs as well as state-of-the-art neural models like glove, longformer and n-shot gpt-4o variants, except for one single rare case in the afd where our llm featurebased model performed comparably with respect to the strongest baseline (0-shot gpt-4o). in addition, the llm feature-based models tend to learn more robust decision rules than those learned by the neural models; the latter has a higher propensity on relying superficial shortcuts. further, interpreting our llm feature-based models can lead to both intuitive findings as well as several unexpected but illuminating discoveries regarding what linguistic factors influence the constructiveness of dialogues. the proposed framework thus serves as a valuable foundational toolkit for dialogue constructiveness research, helping researchers develop models with higher accuracy, robustness and interpretability in dialogue constructiveness assessment.