An Item Response Theory Framework for Persuasion (2022)

Abstract: In this paper, we apply Item Response Theory, popular in education and political science research, to the analysis of argument persuasiveness in language. We empirically evaluate the model’s performance on three datasets, including a novel dataset in the area of political advocacy. We show the advantages of separating these components under several style and content representations, including evaluating the ability of the speaker embeddings generated by the model to parallel real-world observations about persuadability.

## introduction persuasion is the art of instilling in someone a given belief or desire to take a given action. the action can be expressing agreement with the speaker in a debate @xcite , making a donation to a crowdfunding campaign @xcite or non-profit @xcite , or a supreme court ruling @xcite . social psychology frameworks for understanding persuasion, such as the elaboration likelihood model (elm), argue that attributes of successful persuasion fall into three groups: (1) message, the text of the argument; (2) audience; and (3) speaker, the source of the argument. @xcite @xcite . although much attention has been given to studying the text, text in isolation fails to capture how the audiences' prior beliefs and predispositions can affect their response to the same argument. several recent studies have considered all three factors within the context of specific datasets by creating features to represent the audience as a whole or by building separate models for different types of audiences @xcite @xcite . in this paper, we present a broad framework that can represent individual audience members in one model across a diverse set of persuasion tasks. since implementing the elm framework requires separate data about the speaker, audience, and argument, it is difficult to validate empirically. often, we only have access to the observed outcome (e.g. did the person donate money). both the persuadability of the audience and the persuasiveness of the argument are unobserved. motivated by this, we explicitly model a persuasive scenario as a function of latent variables describing the persuadability of the audience and the persuasiveness of the text. our approach is based on item response theory (irt), a framework for modeling the interaction between latent traits and observable outcomes. while these types of models are well known in the context of education @xcite @xcite and politics @xcite , to our knowledge this is the first application of an irt model to study persuasion. using this framework, we model the interaction between the grouped argument and speaker, and the audience, explicitly. the argument and speaker are grouped together because in practice it is hard to separate their effects, especially in the written tasks covered in this study. we explore two variations on the irt framework and apply it to three different persuasion tasks. in addition to two previously studied tasks, we introduce a novel setting related to political advocacy group campaigns, where a recipient is asked by an organization to take a specific action. we evaluate these models with different parameterizations, including style and content features, showing that they are both effective for predicting persuasion, and have the ability to uncover latent characteristics of the audience that were modeled explicitly in previous works. our contributions are as follows: 1) we formalize the use of irt model formulations for persuasion and show the advantages of them over existing approaches, 2) we introduce a new dataset of political advocacy emails, 3) we apply the formulations with style and content features on three persuasion tasks, and 4) we show that the separate latent audience component is interpretable and consistent with external information. all code associated with the paper is available at https: //github.com/akornilo/irt_persuasion. ## item response theory item response theory (irt) represents a set of models that explain an observed outcome based on latent traits. these models are frequently used when an outcome is easily observed, but the factors predicting that model are unobservable. for example, in education an outcome could be a student's answer to an exam question, and the latent predictive traits are a students knowledge and the difficulty of the question; in politics an outcome could be a vote on a bill and the unobservable traits are the legislator's and bill's ideology. crucially, an irt model provides both a prediction of the outcome, and an interpretable measurement of the latent variables. in applying irt to persuasiveness, we can view the audience as having a response to the item, where the item is an argument composed of the speaker and message pair. ## datasets in order to apply the irt framework, an audience member must respond to multiple arguments (and arguments must be observed by multiple audience members). too few responses implies that an audience member's latent value will be driven entirely by the one or two arguments. while not many existing argument mining datasets meet this criteria, we are able to study three diverse settings. additionally, our advocacy task is akin to many real-world settings where users on one-platform are asked to complete an arbitrary task (e.g. a retail mailing list getting users to click on a promotion). ## model features argument analysis is often separated into style and content features @xcite @xcite , with additional categories included for argument quality and task specific properties. since we group the speaker and the argument text together, we combine features representing both as inputs to φ and β. lexicon style features style features represent higher-level properties of words and rhetorical structures. we chose the following sets of such features from lexicons that were commonly used in previous argumentation literature: liwc lexicon of 93 metrics ranging from partsof-speech to thinking styles to emotions (pennebaker et al., 2015); 12 valence, arousal, dominance @xcite ; concreteness @xcite . (these features were shown to be useful for argument quality analysis by @xcite .) argument features developed by @xcite , including necessity, emphasizing, desire, contrasting and rhetorical question; nrc lexicon: word-level level associations for emotions like anger, disgust and fear @xcite ; sentiment and subjectivity: as implemented in the textblob python library. 13 argument text we use tf-idf unigrams to represent the text directly (tuned with respect each 10 those with a lower or higher action rate are unlikely to be illustrative of persuasion characteristics. debate-only speaker features in the debate platform, users can optionally specify a stancefor, against, undecided or no stance -on 48 issues such as abortion, death penalty or gay marriage. these can be viewed as a proxy for the content as users often present arguments that align with their views. advocacy-only org indicator an indicator to account for the large variation in action rate between organizations. additional indicators are used to represent the industry and organization size. advocacy-only appeals using data from @xcite , we built a multi-class classifier to recognize 'emotional', 'logical' and 'credibility' appeals. the classifier was applied at a sentence level to the emails, and features were created for the average and the sum of the scores across the sentences. advocacy-only misc features : the day of the week and time of day have a strong effect on email click rate. 14 we include indicator features for the day of the week and the hour of day. we include an urgency indicator feature, based on a custom list of words indicative of high urgency and timeliness (e.g. "soon", "now", "hurry"). ibm quality @xcite released a dataset of 30k sentence-level arguments with 0-1 quality ratings. unlike our tasks where quality is a latent property, these sentences were assessed for quality directly. we re-implemented the bert-ft model from this paper, using the mace-p score. since these scores were trained on short texts, we apply them to individual sentences in the input text, then use the min, max, average, range, 25th, 50th, and 75th percentiles of these scores. as far as we know, this is the first study to transfer the quality model to longer texts. these features will be grouped with style for the analysis. ## models and results since the editorials corpus is the smallest, we use the simpler rasch parameterization, while the 2pl model is used for the debates and advocacy tasks. 15 each of the models is trained using a regularized binary cross-entropy loss: ## conclusion and future work in this paper, we validate the social psychology frameworks for persuasion using the irt framework to explicitly model the audience and the speaker. our approach lets us analyze how different audience members respond to the same argument, and we show that our representation implicitly learns latent audience features modeled explicitly by other models. we empirically showed several additional insights about persuasion. in the debates and advocacy tasks, the popularity parameter improved performance showing that certain stylistic elements are universally appealing. in the debates task, the audiences' embeddings aligned with their political affiliation, showing that prior beliefs play a strong role in their argument perception. while the background information about the audiences was available for these tasks, we did not need to model it explicitly; as a result this setup allows us to make predictions for audiences who do not report their affiliation. a potential negative side of the models is they may learn latent characteristics of the speaker or