Speech Act Patterns for Improving Generalizability of Explainable Politeness Detection Models (2025)

Abstract: The lack of explainability in state-of-the-art Natural Language Understanding (NLU) classification models has increased interest in developing techniques for improving explainable linear feature-based models (e.g., Logistic Regression/SVM). Politeness detection is a task that exemplifies this interest. While those techniques perform well on the task when applied to data from the same domain as the training data, they lack generalizability and thus fall short when applied to data from other domains. This is due to their reliance on discovering domain-specific word-level features. We introduce a method for improving the generalizability of explainable politeness models by relying on speech act patterns instead of words, leveraging speech act labels assigned by the GPT-4 model. This approach goes beyond the mere words and injects intent into politeness classification models, enhancing their generalizability. Results demonstrate that the proposed method achieves state-of-the-art accuracy in the cross-domain setting among explainable methods, while falling short in the in-domain setting. Our findings illustrate that explainable models can benefit from Large Language Models.

## introduction while recent nlu classification techniques such as bert (i.a. @xcite @xcite and the gpt series (i.a. @xcite @xcite achieve cutting-edge accuracy, their complexity hinders their ability to explain their decisions. this has led to growing interest in developing techniques for improving explainable linear models (e.g., logistic regression/svm). explainable models are essential for transparent adoption of ai (i.a. @xcite @xcite @xcite . an example task for which practitioners developed explainable approaches is politeness detection in text. recent approaches for improving explainable models for this task @xcite rely on discovering fine-grained context patterns of words from a corpus annotated for politeness, and then using the resulting patterns as features to train an explainable model, allowing examination of model coefficients to understand its decisionmaking process. while these features help reduce the gap in performance between explainable models and more complex ones such as bert @xcite when applied to data from the same domain as the training data, they fall short when applied to data from other domains. this is due to their reliance on domain-specific word-level features. we introduce a novel method for improving the generalizability of explainable politeness detection models. the method relies on clustering speech act (e.g. gratitude) textual elements (words, phrases and sentences), and using the resulting clusters as features to improve the performance of explainable models for the task. this allows going beyond words to include the intent of language use. gpt-4 @xcite is used to assign speech act categories to textual elements, and element embeddings are then clustered using aljanaideh et al. (2020)'s algorithm to obtain interpretable features an explainable model can be trained with. results show the proposed method achieves state-of-the-art accuracy on the task of explainable politeness detection in the cross-domain setting, helping reduce the gap in performance between explainable models and a fine-tuned bert @xcite , while falling short in the in-domain setting. the method also automatically discovers subtle politeness cues (e.g., apologizing statements such as i should not have used rollback.). our findings indicate that explainable ai models can be improved by leveraging llms. ## proposed method our goal is to improve the performance of explainable linear models (e.g., logistic regression/svm) for politeness detection. @xcite achieves this by discovering fine-grained context patterns of words using clustering of word embeddings. the resulting clusters are then used as features to train an explainable model for the task. in our method, we rely on speech acts instead of words. specifically, we discover fine-grained clusters of speech acts, and then use those clusters as features to train an explainable model for the task. first, we prompt gpt-4 to extract textual elements (words, phrases, and sentences) associated with pre-defined speech act categories. we then cluster embeddings of each speech act element category using aljanaideh et al. (2020)'s algorithm. this helps discover different ways of using speech acts in (im)polite contexts, even when there is no syntactic overlap between the different elements. for example, i am sorry and apologies are assigned to the same cluster despite having no words in common, since they both contain a form of apology. we describe each of those steps below. ## dataset we use the stanford politeness corpus @xcite . this corpus contains two sub-datasets of requests: one from the wikipedia discussion forum where participants discuss article edits, and another from stack exchange, a qa forum on various topics (e.g. math, farming). the fact that questions span various topics makes it suitable for cross-domain evaluation of politeness cues. the wikipedia portion contains 2,176 requests, while the stack exchange portion contains 3,302 requests. each request is assigned 'polite' or 'impolite' based on the average human annotations. we use the same training/development/test split as aljanaideh et al. (2020). ## results we apply our proposed method to the stanford politeness corpus. below, we describe the results. first, we report the distribution of speech act categories assigned by gpt-4 to obtain insight into the corpus. we then perform qualitative analysis on a sample of discovered clusters. finally, we describe a politeness feature-based explainable classifier that uses the discovered clusters as features and compare its accuracy to existing work on politeness. ## conclusion we introduced a method for improving the generalizability of explainable politeness detection models. the method discovers interpretable features based on speech acts assigned by gpt-4. those features improve over existing methods in the cross-domain setting while falling short in the in-domain setting. this work shows that llms can enhance explainable models.