Unifying Data Perspectivism and Personalization: An Application to Social Norms (2022)

Abstract: Instead of using a single ground truth for language processing tasks, several recent studies have examined how to represent and predict the labels of the set of annotators. However, often little or no information about annotators is known, or the set of annotators is small. In this work, we examine a corpus of social media posts about conflict from a set of 13k annotators and 210k judgements of social norms. We provide a novel experimental setup that applies personalization methods to the modeling of annotators and compare their effectiveness for predicting the perception of social norms. We further provide an analysis of performance across subsets of social situations that vary by the closeness of the relationship between parties in conflict, and assess where personalization helps the most.

## introduction obtaining a single ground truth is not possible or necessary for subjective natural language classification tasks @xcite . each annotator is a person with their own feelings, thoughts, experiences, and perspectives @xcite . in fact, researchers have been calling for the release of data without an aggregated ground truth, and for evaluation that takes individual perspectives into account @xcite . the idea that each annotator has their own view of subjective tasks, and even those previously thought to be objective was introduced by @xcite as data perspectivism. a growth in the interest of this viewpoint has led to the 1st workshop on perspectivist approaches to nlp in 2022. work has examined how to model annotators for subjective tasks and to predict each annotator's label @xcite . modeling annotator perspectives requires the release of corpora that include annotator-level labels rather than aggregated "ground truth" labels. @xcite further recommend releasing data statements that describe characteristics including who is represented in the data and the demographics of annotators. such information is beneficial for raising awareness of the biases in our data. while some corpora contain this information, like those for humor, emotion recognition, and hateful or offensive language, they contain few annotators and no additional information about them @xcite @xcite ). an additional complication for subjective tasks is the fact that different people will interpret text in different ways. what is deemed toxic or offensive depends on who you ask @xcite . there are notable differences in perceived and intended sarcasm @xcite . how one perceives the receptiveness of their own text is different than how others see it @xcite . for such tasks, predicting the label given by third party annotators, without knowing much about them, is not very useful. modeling annotators with personalization methods requires a corpus with many self-reported labels from many annotators, and additional contextual information about them. in this work, we use english textual data in the form of posts from the website, reddit, about social norms from the subreddit /r/amitheasshole (aita). as shown in figure 1 , users of this online community post descriptions of situations, often involving interpersonal conflict, and ask other users to judge whether the user acted wrongly in the situation or not. the judgements from these users constitute our labels, and their authors are the set of annotators (and we refer to them as such for the remainder of the paper), which allows us to explore methods to model annotators at a larger scale. we explore methods of personalization to model these annotators and examine how the ef-fectiveness of our approach varies with the social relation between the poster and others in the described situation. we further provide an analysis of how personalization affects demographic groups and how performance varies across individuals. our contributions include (1) a discussion of the relation between data perspectivism and personalization, (2) a novel problem setting involving a recently collected dataset with unique properties allowing us to explore these concepts for annotator modeling, and (3) a novel comparison of contemporary personalization methods in this setting. ## formulation we formalize our task in terms of the textual data points, their authors, annotators, and the annotations they provide. a poster, u, makes a post, p, which is then commented on by an annotator, with id a, who provides a comment, c a,p , and a label, or verdict, v a,p . since we are modeling annotators, u is not important to us, except that u ̸ = a within the same post. each post p has many comments c p , though this is not strictly necessary for our purposes, it does help reveal the subjectivity of the task. importantly, each annotator, a, has many comments, c a . in our case, the comment c a,p i written by annotator a on the i-th post p i , is linked to a single v a,p i , though one could gather these from separate sources, and doing so may be necessary for other corpora. the subjective nature of the task and its evaluation comes from the assumption that annotators provide different verdicts for a post. work on annotator modeling attempts to estimate the probability of a verdict given the post and annotator, p(v a,p |a, p). this is in contrast to predicting what an individual's language means, p(v a,p |a, p, c a,p ), which we refer to as a personalized classification task. importantly, we make this distinction because personalization has historically focused on predicting a label assigned to an individual's text in a particular context (e.g. the sentiment of a review), whereas work on modeling annotators focuses on the label an individual would assign in that context. there is often no information about annotators or only an id is known. a few works on annotator modeling include extra information about the annotator, t . this information can be defined in various ways (see §3.3). in this work, we use a collection of other texts from the annotator (see §6). to the best of our knowledge, our formulation of t is novel in that it allows the application of previously developed methods for personalization to the task of annotator modeling. importantly, we are predicting how the annotator will label the post, p(v a,p |a, p, t ), not how to interpret their text. for other work that has attempted to interpret verdicts, p(v a,p |a, c a,p ), refer to §3.1. ## related work in our work, we refer to users as social media users. within aita, we refer to the poster as the user who originally made the post, and annotators as those who commented on the post. both posters and annotators are authors of their respective posts and comments. ## clustering for the purpose of this work, we cluster the types of situations in our dataset as described at @xcite . to obtain this clustering, a graph with nodes representing the posts (situations) and edges weighted by their similarity is constructed. the situations are represented by their post titles or full text, and the relevant text is embedded with sentence-bert (sbert, @xcite ) to form the set of nodes of the graph. the edges are created for each pair of situations using the cosine similarity normalized to [0, 1], resulting in a fully connected weighted graph. furthermore, the louvain clustering algorithm is applied to obtain the clusters, which maximizes the modularity of the weighted graph @xcite . afterwards, an algorithm is applied to prune the graph. the n% lowest weighted edges of the graph in steps of 10% are dropped, and louvain clustering is applied at each step. then, for each pair of graphs that differ only by one step, that is having x% and (x+10)% of edges removed, the adjusted rand index is determined between those graphs to find a range where the clustering is least affected by edge removal and thus persistent. this method leads to a 30% cutoff for full texts and 40% for situations. in both cases, this resulted in 3 clusters which were then labeled by two annotators. ## methodology as described in §2, we are attempting to model p(v a,p |a, p, t ), or the probability of a verdict given by an annotator of a post, with additional information about that annotator, t . in our work, we define @xmath0,q |q ̸ = p, ∀@xmath1, where p is the set of posts the annotator has commented on. in the following subsections, we describe how we encode text and the personalization methods we use to model t . ## experiments we experiment with four personalization methods for annotator modeling and two situation text baselines for a secondary task of personalized verdict interpretation. sbert (text only interpretation model): as our base model, we finetune sbert on the binary task of predicting the verdict, given the comment and the situation title. judgebert (text only interpretation model): we compare our personalized models to judge-bert @xcite , a recent model that was developed to study moral judgements, and reported the highest performance of the models discussed in §3. though our novel task setup does not have an existing baseline to directly compare to, this comparison, which does use the verdict text, serves as a point of reference. ## analysis & discussion here we attempt to understand how our models perform with respect to individuals, demographic groups, available data, and the type of task. ## conclusion we experimented with a recently constructed dataset with unique properties that allowed us to explore annotator modeling using personalization methods. we filtered and augmented the data to obtain a set suitable for our experiments which contains self-identified demographics. our approach unifies calls for data perspectivism with approaches developed for large volumes of user information. within this setting we are also able to provide a comparison of recent personalization methods. overall, we found that averaging embeddings provided a strong and relatively simple approach, though each model has its strengths, the authorship attribution and graph attention networks were consistently high performing across splits, while for the situation split, annotator-level accuracy was highest with the priming approach. these methods outperform the common approach of representing authors with a single id. as a secondary result, we found that personalized methods significantly outperformed previous work and text only baselines on the task of interpreting verdicts. we then showed an analysis of performance across tasks, showing that the closer the relationship between parties in conflict, the more personalization helps model the judgements annotators will give. we further showed performance across demographics, showing that our methods appear unbiased in this regard, despite being trained with more data from females than males. we revealed a correlation between the amount of data from an annotator during training, and it's impact on personalization, showing that more data generally, but not always, helps. we hope that our formalization of this task provides a path for future work in this direction and our insights for the task of predicting judgements of social norms provide meaningful first steps. our code is publicly available at https://github.com/caisalab/perspectivism-personalization.git