Performance of BERT on Persuasion for Good (2021)

Abstract: We consider the task of automatically classifying the persuasion strategy employed by an utterance in a dialog. We base our work on the PERSUASION-FOR-GOOD dataset, which is composed of conversations between crowdworkers trying to convince each other to make donations to a charity. Currently, the best known performance on this dataset, for classification of persuader’s strategy, is not derived by employing pretrained language models like BERT. We observe that a straightforward fine-tuning of BERT does not provide significant performance gain. Nevertheless, nonuniformly sampling to account for the class imbalance and a cost function enforcing a hierarchical probabilistic structure on the classes provides an absolute improvement of 10.79% F1 over the previously reported results. On the same dataset, we replicate the framework for classifying the persuadee’s response.

## introduction with the advancement of artificial intelligence, there has been a tremendous rise in its usage in the daily lives of people. birth of conversational agents has made life a lot easier for organizations looking to achieve certain tasks. these agents, like as mentioned in @xcite @xcite , are goaloriented, i.e., they try to engage users in meaningful conversations and thereby aim to achieve their tasks. at times, they require different strategies of persuasion in order to mould people into their way of thinking, thereby changing their specific attitude or behaviour @xcite . @xcite proposed the foundation on building an automatic personalized persuasive dialogue system. they created the persuasion-for-good dataset, which is composed of conversations between crowdworkers trying to convince each other to make donations to a charity. they annotated the utterances with persuasive strategy labels and proposed a baseline method for persuasive strategy classification. until recently, the dominant prototype in approaching any natural language processing tasks has been to focus on designing neural network architectures, using task specific data and word embeddings such as glove @xcite . the nlp community is witnessing a paradigm shift towards pre-trained deep language representation model which achieves the sota in question answering, sentiment analysis and other nlp tasks. bidirectional encoder representations from transformers (bert) @xcite represents one of the latest developments in this field. it surpasses its predecessors, elmo @xcite and gpt @xcite by a significant margin on numerous nlp tasks. there is not much literature on exploring bert for tasks related to persuasive dialogues. in this work, we introduce a bert-based approach to automatically classify the persuasion strategy employed by an utterance in a dialog. we also use the same approach to classify the type of the response of the persuadee's utterance. we base our work on the persuasion-for-good dataset. since the amount of annotated dialogues in this dataset are very less, we experiment to evaluate the efficacy of pretrained bert in achieving better performance for the said task. the main contribution of this work is : 1. creating a bert-based hierarchical classification setup for classification of persuader's strategy . 2. creating a benchmark setup for the persuadee's response classification 3. additional analysis for the dataset introduced by @xcite . the baseline performance for strategy classification on persuasion-for-good is an f1 score of 59.6% and 74.8% accuracy @xcite . we observe that a straightforward fine-tuning of bert does not provide significant performance gain: 60.60% f1 and 75.85% accuracy. never-theless, non-uniformly sampling to account for the class imbalance does improve performance to 68.1% f1 and 77.69% accuracy. we further employed a cost function which enforces a hierarchical probabilistic structure on the classes, namely a utterance is persuasive or not, and if persuasive, the strategies further belong to coarser classes of appeal or inquiry. this step improves the performance to 70.39% f1 and 79.50% accuracy, which is an absolute improvement of 10.79 f1 over previously reported results. the remainder of this paper is organized as: we review the related work in section 2. in section 3 we conduct data analysis. in section 4 we describe our methodology. we analyze the results and present our observations in section 5. finally, we summarize the key conclusion in section 6.