Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations (2024)

Abstract: We present a generalizable classification approach that leverages Large Language Models (LLMs) to facilitate the detection of implicitly encoded social meaning in conversations. We design a multi-faceted prompt to extract a textual explanation of the reasoning that connects visible cues to underlying social meanings. These extracted explanations or rationales serve as augmentations to the conversational text to facilitate dialogue understanding and transfer. Our empirical results over 2,340 experimental settings demonstrate the significant positive impact of adding these rationales. Our findings hold true for in-domain classification, zero-shot, and few-shot domain transfer for two different social meaning detection tasks, each spanning two different corpora.

## introduction the former embodies the idea of linguistic agency, the deliberate choices people make to protect their identity @xcite or to accomplish social goals @xcite . the latter encompasses involuntary cues that signal dispositions, like personality @xcite , attitude @xcite , or emotion @xcite , or psychological conditions, like mental illness @xcite , since social meaning is subtly encoded, traditional classification models often over-fit to contextspecific linguistic elements that correlate with these subtle cues within context. consequently, this makes transfer to unseen domains especially challenging. for example, the same strategy to resist being persuaded would manifest in different ways depending on whether one is negotiating the price of a commodity, or one is hesitating donating to charity @xcite . in this work, we propose a generalizable framework that leverages large language models (llms) for detecting different kinds of social meaning in conversations. we systematically investigate the generation of "rationales" by llms, that are designed to break through the opaque surface form of the conversation's text and make the social cues more transparent. while rationales have been utilized previously, to facilitate reasoning @xcite , or to explain model predictions @xcite , we use rationales to refer to the elicited social meaning, i.e., why and how an utterance was conveyed in dialogue. our empirical study examines the role of augmenting rationales for two specific social meaning detection tasks: (i) resistance strategies (res), which aligns with intentional and purposeful communication, and (ii) emotion recognition (erc), which is characterized by habitual and subconscious responses. for each of these tasks, the evaluation is conducted over two separate corpora (different domains), but the same social meaning detection task. and thus we present results both for the in-domain (id) and transfer (tf) settings. we illustrate in figure 1 that baseline models performed significantly worse than their rationale-augmented counterparts for both tasks and settings. our contributions are as follows : we provide the datasets augmented with rationales and code as public resources to encourage future research, especially for the purpose of developing open-source solutions that achieve the same functionality as the proprietary llms that perform best in our studies. we also make our code and data publicly available here foot_0 . ## prompting framework in this section, we propose a prompting framework to generate rationales that can capture the underlying social meaning and assess their validity. we showcase our prompting framework in figure 2 . ## qualitative analysis having demonstrated the efficacy of rationales to facilitate understanding of social meaning in dialogue, we do a deep dive on their utility, namely where do rationales help and why. we investigate the impact rationales have on individual task labels or strategies in id. for each dataset, we consider the combination of model and rationale pair with the highest id performance in we highlight examples where adding rationales were consistently better in rationales perform worse for erc: the cases where the model mispredicts can be linked to the specific language usage. for example, the utterance in friends "what the hell happened on that beach?!" is erroneously interpreted as anger possibly due to "what the hell." likewise, for the utterance "i'm just worried," in iemocap, the rationales express a sense of anxiety or uncertainty from "worried" misleading the prediction as "other" than "sadness." rationales perform better for res: for res, the integration of rationales notably enhances performance for "counter argumentation" and "hesitance." e.g., in the cb dataset, for the utterance "but how about 180 since i'm the one picking it up and with its one handle missing?", the rationale accurately identifies the buyer's intention to propose a reduced price due to the item's missing handle, and thus aligns with counter argumentation. furthermore, for p4g, "when finished with this task i will be sure to check the website," the rationales portray the speaker's implied conditional interest, indicating hesitance as the action is deferred until task completion. rationales perform worse for res: conversely, the model's performance for the "source derogation" strategy is less effective. a typical example is "perhaps a link to an organization or other agency that rates major charities would be more helpful" for p4g. here, the rationales inaccurately interpret the statement as a mere suggestion for a more efficient information source, and fail to detect the speaker's skepticism about the organization's credibility. we posit that this misprediction is linked to llm's tendency to generate responses with a positive connotation, leading to a misinterpretation of critical tones as constructive suggestions. this results in erroneous labeling as "information inquiry" indicating a request for additional information, or "counter argumentation," which suggests an alternative factual proposition. while we note that overall rationales facilitate transfer, the gains observed are not symmetric. specifically, we observe higher gains for the less frequent classes in the target dataset, such as the emotion "fear" on friends and "source derogation" and "self pity" classes on the p4g dataset. ## conclusion and future work we present a generalizable framework that leverages machine-generated rationales from llms to deduce the underlying social meaning embedded in conversations. we observe that augmenting pretrained models with the generated rationales significantly improves performance over the baseline across multiple datasets for both the tasks of emotion recognition and detecting resisting strategies. the gains are pronounced during cross-domain transfer across both zero-shot and few-shot settings thereby highlighting the generalizability of our approach. while our current work place emphasis on domain adaptation, we believe the proposed approach is generalizable to new social meaning detection tasks (persuasion, empathy, argumentation) which we defer for future work. furthermore, as opposed to leveraging an llm, we intend to deploy or instruct-tune smaller models that can generate these rationales @xcite .