S ty LE x: Explaining Style Using Human Lexical Annotations (2023)

Abstract: Large pre-trained language models have achieved impressive results on various style classification tasks, but they often learn spurious domain-specific words to make predictions (Hayati et al., 2021). While human explanation highlights stylistic tokens as important features for this task, we observe that model explanations often do not align with them. To tackle this issue, we introduce StyLEx, a model that learns from human annotated explanations of stylistic features and jointly learns to perform the task and predict these features as model explanations. Our experiments show that StyLEx can provide human like stylistic lexical explanations without sacrificing the performance of sentence-level style prediction on both in-domain and out-of-domain datasets. Explanations from StyLEx show significant improvements in explanation metrics (sufficiency, plausibility) and when evaluated with human annotations. They are also more understandable by human judges compared to the widely-used saliency-based explanation baseline.

## introduction people use style as a strategic choice for their personal or social goals in communications, making style analysis a long-studied field in nlp @xcite @xcite . while large language models have obtained state-of-the-art results on many nlp tasks, they have been shown to overfit to spurious correlations in data across several datasets @xcite @xcite . @xcite found a phenomenon in style classification tasks where the model's word-level explanation do not align with human's stylistic cues (stylistic cues are words that signify the style of a text). for instance, words such as "performances" and "wrench" .. top notch .. in figure 1 are marked as important cues for sentiment by a saliency method. however, they are different from words that humans perceive as essential features for predicting the style ("top," "notch"). prior research in style have developed stylistic lexicon dictionary to identify the style of a text, such as sentiment or emotion, and in-turn incorporated them for style classification tasks @xcite @xcite . while lexiconbased matching methods @xcite provide interpretability for the task, they lack coverage and do not incorporate the context for prediction. on the other hand, current large scale models like bert @xcite are effective at style classification. however, their explanations often reveal that the model do not rely on the stylistic words to make the prediction. in this work, we hypothesize that leveraging stylistic lexica along with the effectiveness of a large scale model like bert can not only predict style but also provide meaningful explanations that align with human style explanations. towards this, we introduce stylex, a style classification model that jointly learns to align humanannotated stylistic cues as explanations and then predict the style of the overall sentence based on the cues (figure 1 ). stylex uses a semi-supervised approach to expand the stylistic words from a handful number of human-annotated stylistics words. first, we train stylex using existing small human stylistic word annotation from @xcite . then, we obtain predicted stylistic words on the larger benchmarking style datasets and retrain stylex on this expanded data to predict both the sentence's style label and the stylistic lexical explanation. in this study, we show that for both in-domain and out-of-domain data, stylex not only shows competitive classification performance with bertbased model, but also generate stylistic lexical explanations that have higher alignment with human explanations. in terms of explanation quality, stylex surpasses the baseline method across multiple explanation metrics. for the sufficiency metric, we improve upon the baseline by 14.12% on the average. for plausibility, stylex's lexical explanations correlate highly with human lexical annotation ground truth with an average pearson's r correlation score of 44% compared to the baseline's correlation score of 3.9%. finally, we found that 72.5% of stylex's explanations are more preferred by human judges compared to the baseline. ## style explanation evaluation we investigate stylex's explanations if they are sufficient, plausible, and understandable following previous works @xcite @xcite . jacovi and goldberg (2020) define that a faithful interpretation represents a model's reasoning process. to evaluate whether stylex's explanations are faithful, we run a sufficiency test that evaluates whether the model explanations alone are highly relevant for predicting the label @xcite . meanwhile, we measure plausibility to examine whether the explanation is agreeable to humans @xcite . finally, understandability measures if a user is able to understand model explanations @xcite . for investigating sufficiency and plausibility, we run automatic metrics. to assess understandability, we ask human judges to choose the explanation that can be better understood by a non-expert between stylex and the integrated gradients (ig) method. ## related work styles in nlp research on style in nlp has addressed various tasks including style classification @xcite , style transfer @xcite , style and content disentanglement @xcite , and multiple style analysis @xcite . this work focuses on understanding stylistic variation in style classification. style classification models often produce spurious features @xcite @xcite , motivating us to leverage stylistic variation from human perspectives to distinguish between stylistic words and content words. past work have used stylistic lexica for classification @xcite , but in this work, we fine-tune the language model to generate these lexica and use them to help style prediction. our work is most closely related to @xcite , but they do not develop any new models to use the human perception scores as explanations. moreover, while linguistics styles can cover an author's writing style or figurative language, we limit our study to high-level style as used in kang and @xcite . explainable nlp heat maps generated from attention values from the models @xcite are widely used as an interpretability tool, but these attention maps are often unfaithful and unreliable (jain and wallace, 2019; @xcite @xcite . saliency maps computed via gradients offer an alternative @xcite @xcite . annotating explanations as rationales (part of input) @xcite through expert annotations @xcite is widely used to model explanations in nlp when external annotations are available. another class of inherently interpretable models aims to optimize model explanations without any external annotations @xcite @xcite . our work is similar in spirit to the rationale approaches @xcite but focuses on understanding style attributes in text and computationally modeling them based on human annotation of the important words. ## conclusion we proposed stylex, a style classification model for learning stylistic variations through lexical explanation. with only 500 sentences with wordlevel style annotation, we find improvement in both classification and explanation. compared to the commonly-used integrated gradient method, stylex's explanations are more accurate for model prediction, more consistent with human-found stylistic words from existing datasets and lexicon dictionaries, and better understandable by human judges, without sacrificing task performance on both in-domain and out-of-domain datasets. future work our approach opens up future work on human-centered lexical explanation for correcting the spurious behavior of nlp models and for better explaining linguistic styles. we plan to investigate collecting more human lexical annotations to more accurately model stylistic variation, especially with larger pretrained language models. broader usage of stylex in providing stylistic cues will be applicable to lexical style and content disentanglement @xcite , counterfactual data augmentation for style-related tasks @xcite , and stylistic paraphrasing @xcite . limitations our work has some limitations, mostly stemming from the size and nature of the human-annotated data. the training data (500 sentences) from hummingbird is quite small to train deep learning models. however, our work shows that with just 500 sentences we could achieve a huge improvement in interpretability as well as a slight improvement in ood performance, using semi-supervised training. moreover, there is sparsity of stylistic words in the sentences. we also found that some stylistic words have various scores of human perception; capturing such subtle stylistic words is difficult. an interesting future work would be to handle these problems of sparsity and subtlety. we also notice that hummingbird is annotated by people residing in the united states. thus, their perception of styles may not reflect the perception of those with different cultural backgrounds. nevertheless, stylex can be applied to any dataset training with similar human lexical annotations, and not limited to hummingbird.