From Toxicity in Online Comments to Incivility in A merican News: Proceed with Caution (2021)

Abstract: The ability to quantify incivility online, in news and in congressional debates is of great interest to political scientists. Computational tools for detecting online incivility for English are now fairly accessible and potentially could be applied more broadly. We test the Jigsaw Perspective API for its ability to detect the degree of incivility on a corpus that we developed, consisting of manual annotations of civility in American news. We demonstrate that toxicity models, as exemplified by Perspective, are inadequate for the analysis of incivility in news. We carry out error analysis that points to the need to develop methods to remove spurious correlations between words often mentioned in the news, especially identity descriptors and incivility. Without such improvements, applying Perspective or similar models on news is likely to lead to wrong conclusions, that are not aligned with the human perception of incivility.

## introduction surveys of public opinion report that most americans think that the tone and nature of political debate in this country have become more negative and less respectful and that the heated rhetoric by politicians raises the risk for violence @xcite . these observations motivate the need to study (in)civility in political discourse in all spheres of interaction, including online @xcite , in congressional debates @xcite and as presented in news @xcite . accurate automated means for coding incivility could facilitate this research, and political scientists have already turned to using off-the-shelf computational tools for studying civility @xcite @xcite . computational tools however, have been developed for different purposes, focusing on detecting language in online forums that violate community norms. the goal of these applications is to support human moderators by promptly focusing their attention on likely problematic posts. when studying civility in political discourse, it is primarily of interest to characterize the overall civility of interactions in a given source (i.e., news programs) or domain (i.e., congressional debates), as an average over a period of interest. applying off-the-shelf tools for toxicity detection is appealingly convenient, but such use has not been validated for any domain, while uses in support of moderation efforts have been validated only for online comments. we examine the feasibility of quantifying incivility in the news via the jigsaw perspective api, which has been trained on over a million online comments rated for toxicity and deployed in several scenarios to support moderator effort online foot_0 . we collect human judgments of the (in)civility in one month worth of three american news programs. we show that while people perceive significant differences between the three programs, perspective cannot reliably distinguish between the levels of incivility as manifested in these news sources. we then turn to diagnose the reasons for perspective's failure. incivility is more subtle and nuanced than toxicity, which includes identity slurs, profanity, and threats of violence along other unacceptable incivility. in the range of civil to borderline civil human judgments, perspective gives noisy predictions that are not indicative of the differences in civility perceived by people. this finding alone suggests that averaging perspective scores to characterize a source is unlikely to yield meaningful results. to pinpoint some of the sources of the noise in predictions, we characterize individual words as likely triggers of errors in perspective or sub-error triggers that lead to over-prediction of toxicity. we discover notable anomalies, where words quite typical in neutral news reporting are confounded with incivility in the news domain. we also discover that the mention of many identities, such as black, gay, muslim, feminist, etc., triggers high incivility predictions. this occurs despite the fact that perspective has been modified specifically to minimize such associations @xcite . our findings echo results from gender debiasing of word representations, where bias is removed as measured by a fixed definition but remains present when probed differently @xcite . this common error-treating the mention of identity as evidence for incivility-is problematic when the goal is to analyze american political discourse, which is very much marked by us-vs-them identity framing of discussions. these findings will serve as a basis for future work in debiasing systems for incivility prediction, while the dataset of incivility in american news will support computational work on this new task. our work has implications for researchers of language technology and political science alike. for those developing automated methods for quantifying incivility, we pinpoint two aspects that require improvement in future work: detecting triggers of civility overprediction and devising methods to mitigate the errors in prediction. we propose an approach for a data-driven detection of error triggers; devising mitigation approaches remain an open problem. for those seeking to contrast civility in different sources, we provide compelling evidence that state-of-the-art automated tools are not appropriate for this task. the data and (in)civility ratings would be of use to both groups as test data for future models for civility prediction foot_1 . ## data collection for incivility in news we study the following american programs: pbs newshour, msnbc's the rachel maddow show and hannity from fox news. for brevity, we use the network to refer to each source: pbs, msnbc, and fox, in the following discussions. these sources are roughly representative of the political spectrum in american politics, with newshour being left-center and msnbc and fox with strong left and right bias respectively. these are generally one-hour shows, with about 45 minutes of content when commercial breaks are excluded. the transcripts, which provide speaker labels and turns, are from february 2019. we take only the days when all three shows aired, and we had transcripts, for a total of 51 transcripts. we analyze the programs on two levels with the help of paid research assistants paid $15 per hour. all of them are undergraduate students in non-technical majors at the university of pennsylvania. in pass i, we ask a single rater to read through a transcript and identify speaker turns that appear particularly uncivil or notably civil. we characterize the programs according to the number of uncivil segments identified in each transcript. 3 after that, in pass ii, a research assistant chose a larger snippet of about 200 words that includes the selection identified as notably uncivil (and respectively civil), providing more context for the initially selected speaker turns. we also selected several snippets of similar length at random but not overlapping with the civil and uncivil snippets. each of these snippets, 219 it total, was then rated for perceived civility by two raters, neither of whom participated in the initial selection of content. the raters did not know why a snippet that they were rating was selected. we choose to annotate such snippets, corresponding to about a minute of show content, to ensure sufficient context is available to make meaningful judgments about the overall tone of presentation. the composite civility score is then obtained by first reversing the ratings to account for the alter-nations of the ends for each dimension, such that all ratings result in small ratings for the civil ends of the scale (polite, friendly, cooperative, calm) and high values for the uncivil ends of the scales. the four scores were averaged for each annotator. finally, the scores for the two annotators are averaged to obtain a civility score for each snippet, ranging between 1 and 10, where 1 is the most civil and 10 is the most uncivil possible. ## annotator agreement here we briefly discuss the annotator agreement on the perceived incivility. we characterize the agreement on the transcript level in terms of the civil and uncivil speaker turns flagged in pass i and on the text snippet level in terms of correlations and absolute difference in the scores assigned by a pair of raters in pass ii. pass i selection of turns was made by one person, but we are still able to glean some insights about the validity of their selection from analyzing the ratings of pass ii snippets. the pass i selection can be construed as a justification for the score assigned in pass ii, similar to other tasks in which a rationale for a prediction has to be provided @xcite . figure 2 shows the distribution of scores for the 200-word snippets that were selected around the initial 40-50-word speaker turns deemed notably civil, uncivil or that were randomly chosen. the distribution is as expected, with segments including uncivil turns almost uniformly rated as uncivil, with civility score greater than 5. according to our scale, a score of 5 would correspond to borderline civil on all dimensions or highly uncivil on at least one. only three of the 105 snippets selected around an uncivil turn got scores a bit under 5: the remaining snippets including a rationale for incivility, were rated as uncivil with high consistency by the independent raters in pass ii. in pass ii, each pair of annotators rated a total of 37 or 36 segments. correlations between the composite ratings are overall high, ranging from 0.86 to 0.65 per pair. for only one of the six pairs, the correlation is below 0.75. the absolute difference in composite incivility scores by the two independent annotators is about 1 point. ## predicting incivility in american news with the corpus of real-valued civility in the news, we can now compare sources according to the perception of people with those provided by perspective. we perform the analysis on the transcript level and the snippet level. on the transcript level, we use the number of uncivil utterances marked in pass i as the indicator of incivility. for the snippet level, we use the average composite civility ratings by the raters in pass ii. for automatic analysis, we obtain perspective scores for each speaker turn as marked in the transcript and count the number of turns predicted as having toxicity 0.5 or greater. for snippets, we obtain perspective scores directly, combining multiple speaker turns. figure 3 gives the scatter plot of civility scores per segment, from raters and perspective. the plot reveals that perspective is not sensitive enough to detect any differences in levels of incivility for human rating lower than six. for the higher levels of incivility, perspective scores also increase and have better differentiation. however, the snippets from msnbc rated as uncivil by people receive low scores. we verified that these segments are indeed uncivil, but in a sarcastic, indirect way. portions of the turns can be seen in there is only one snippet with perspective toxicity score over 0.5 for the civil to borderline civil segments from the news shows; this indicates it has good precision for binary identification of civil content. perspective's performance in detecting binary incivility (for snippets with ratings greater than 5) is mixed. the recall for incivility is not that good, with some of these snippets receiving low toxicity scores. the trend of increasing perspective score with increasing human-rated incivility is observed mostly on segments from fox. the incivility in fox appears to be more similar to that seen in online forums, with name-calling and labeling of individuals. some examples can be seen in however, when characterizing shows by incivility, detecting utterances that may be problematic is not of much interest. the goal is to characterize the show (daily, weekly, monthly) overall. for this, we inspect the average incivility per show on the transcript and segment level (see on the transcript level, the presence of incivility in pbs is not statistically significant. the raters chose 0.29 (fewer than one) uncivil utterances from pbs from all shows on the 17 days we study, compared with 4.5 per show for fox. the 95% confidence interval for the mean for uncivil utterances per show covers zero for pbs, so it is not statistically significant. the lower end of the 95% confidence intervals of the mean transcript-level incivility in fox and msnbc is greater than zero, indicating consistent incivility in these programs. the segment-level analysis of civility ratings reveals the same trend, with a one-point difference between pbs and msnbc and two points between msnbc and fox. all of these differences are statistically significant at the 0.01 level, according to a two-sided mann-whitney test. the automated analyses paint a different picture. on the transcript level, fox is overall the most uncivil, with about 6 speaker turns per predicted to be toxic, with a perspective score greater than 0.5. pbs appears to be the next in levels of incivility, with more than one toxic turn per transcript. for both of these shows, incivility is overpredicted, and many of the segments predicted as uncivil are civil according to the human ratings. msnbc is predicted to have fewer than one toxic turns per transcript, under-detecting incivility. on the segment level, fox is again assessed as the most uncivil, and pbs again appears to be more uncivil than msnbc. on the segment level, the differences are statistically significant. perspective incorrectly characterizes pbs as significantly more uncivil than msnbc. the correlation between human and perspective incivility scores is 0.29 on the transcript and 0.51 on the segment level. overall for broadcast news, perspective cannot reproduce the incivility perception of people. in addition to the inability to detect sarcasm/snark, there seems to be a problem with over-prediction of the incivility in pbs and fox. in the next section, we seek to establish some of the drivers of over-prediction errors, characterizing individual words as possible triggers of absolute or relative over-prediction of incivility. ## incivility over-prediction prior work has drawn attention to the fact that certain words describing people are incorrectly interpreted as triggers of incivility by perspective, leading to errors in which a perfectly acceptable text segment containing the words would be predicted as toxic @xcite . their analysis, similar to other work on bias in word representations, starts with a small list of about 50 words to be analyzed in an attempt to find toxicity over-prediction triggers. in our work, we seek to apply the same reasoning with the same goals, but in a more data-driven manner, without having to commit to a very small list of words for analysis. given our text domain of interest (news) and the desiderata to characterize sources rather than individual text segments, we also find sub-errors, or words that do not lead to errors in toxicity prediction but have much higher than average toxicity associated with them compared to other words. ideally, we would like to test the full vocabulary of a new domain for (sub-)error triggers, but methods for doing so do not exist and may not be practical when the vocabulary is too large. for this reason, we sample words in a way informed by the target application, choosing words that con- ## error trigger words for perspective we consider a word to be an error trigger if at least one of the templates has a toxicity score of 0.5 or greater from perspective. below is the full list of error trigger words from the news shows. we informally organize them into categories to facilitate inspection. identity african-americans, anti-semites, anti-white, antiwomen, blackkklansman, feminism, feminist, gay, homophobia, homophobic, homophobic, islam, islamic, jew, lgbt, muslim, women violence and sex annihilation, assassinated, beheaded, die, kill, killed, killing, murdered, shooting; intercourse, pedophiles, pornography, prostitution, rape, raped, rapist, rapists, sex, sexist, sexual, sexual, sexually, sodomized ambiguity dick, dirty, garbage, rats ## sub-error triggers sub-error triggers of incivility are words that are not offensive and for which perspective returns a toxicity score below 0.5 for each of the five templates when the word is plugged in the template. the error triggers we discussed above lead to actual errors by perspective for its intended use. the suberror triggers are in the acceptable level of noise for the original purpose of perspective but may introduce noise when the goal is to quantify the typical overall civility of a source. to establish a reference point for the expected average template civility of non-offensive words, we sample 742 words that appeared in at least 10 speaker turns (i.e., were fairly common) and ap-peared in speaker turns in the news shows that received low toxicity predictions from perspective, below 0.15. these were part of the list we classify as offensive or not. no word in this reference sample was labeled as offensive. the average template score for the reference sample is 0.09, with a standard deviation of 0.01. we define sub-error triggers to be those whose average template toxicity score is two standard deviations higher than the average in the reference list. there are 325 non-offensive words that meet the criteria for sub-error triggers. they are shown in a sizeable percentage of sub-error triggers are related to identities of people-gender, age, religion, country of origin. oddly, a number of childdescribing words appear in the list (baby, child, kid, toddler). there are also several personal names in the list, which indicates spurious correlations learned in perspective; names by themselves can not be uncivil. second person pronouns (you, your) and thirdperson feminine pronouns (her, she) are sub-error triggers. the second-person pronouns are likely spuriously associated with toxicity due to overrepresentation in direct toxic comments directed to other participants in the conversation. similarly, the association of female pronouns with toxicity is likely due to the fact that a large fraction of the indirect toxic comments online are targeted to women. regardless of the reasons why these words were associated with incivility by perspective, the vast majority of them are typical words mentioned in the news and political discussions, explaining to an extent why perspective is not able to provide a more segments from pbs contain uncivil words, compared to msnbc. ## conclusion the work we presented was motivated by the desire to apply off-the-shelf methods for toxicity prediction to analyse civility in american news. these methods were developed to detect rude, disrespectful, or unreasonable comment that is likely to make you leave the discussion in an online forum. to validate the use of perspective to quantify incivility in the news, we create a new corpus of perceived incivility in the news. on this corpus, we compare human ratings and perspective predictions. we find that perspective is not appropriate for such an application, providing misleading conclusions for sources that are mostly civil but for which people perceive a significant overall difference, for example, because one uses sarcasm to express incivility. perspective is able to detect less subtle differences in levels of incivility, but in a large-scale analysis that relies on perspective exclusively, it will be impossible to know which differences would reflect human perception and which would not. we find that perspective's inability to differentiate levels of incivility is partly due to the spurious correlations it has formed between certain non-offensive words and incivility. many of these words are identity-related. our work will facilitate future research efforts on debiasing of automated predictions. these methods start off with a list of words that the system has to unlearn as associated with a given outcome. in prior work, the lists of words to debias came from informal experimentation with predictions from perspective. our work provides a mechanism to create a data-driven list that requires some but little human intervention. it can discover broader classes of bias than people performing ad-hoc experiments can come up with. a considerable portion of content marked as uncivil by people is not detected as unusual by perspective. sarcasm and high-brow register in the delivery of the uncivil language are at play here and will require the development of new systems. computational social scientists are well-advised to not use perspective for studies of incivility in political discourse because it has clear deficiencies for such application.