NLP Systems That Can’t Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps (2024)

Abstract: The use of words to convey speaker’s intent is traditionally distinguished from the ‘mention’ of words for quoting what someone said, or pointing out properties of a word. Here we show that computationally modeling this use-mention distinction is crucial for dealing with counterspeech online. Counterspeech that refutes problematic content often mentions harmful language but is not harmful itself (e.g., calling a vaccine dangerous is not the same as expressing disapproval of someone for calling vaccines dangerous). We show that even recent language models fail at distinguishing use from mention, and that this failure propagates to two key downstream tasks: misinformation and hate speech detection, resulting in censorship of counterspeech. We introduce prompting mitigations that teach the use-mention distinction, and show they reduce these errors. Our work highlights the importance of the use-mention distinction for NLP and CSS and offers ways to address it.

## introduction the use-mention distinction is the difference between using words (bananas have a peel) and mentioning them ("bananas" has 7 letters, or dan said "bananas"). the distinction has long been important in the philosophy of language @xcite , where discussions date back to @xcite and @xcite . the ability to correctly make this distinction is particularly relevant to dealing with problematic language online. while detecting problematic text has become a standard task in nlp, there has been less consideration of text that might mention harmful content without directly using it. mentioning problematic content is critical to language like counterspeech that challenges or opposes hateful or misleading narratives ( intention state hate speech challenge hate speech speaker source use vs. mention jews are born greedy. the idea that jews are born greedy arises from medieval stereotypes, that's the only truth. true use mention to challenge hate speech detection detection state misinformation challenge misinformation speaker it's not a vaccine it is gene therapy! it changes your dna which you cannot change back your claim that vaccine is gene therapy is inaccurate. multiple independent fact-checking [...] ## discussion implications for content moderation existing datasets of harmful content are typically collected by sampling keywords that co-occur with harmful content and performing annotation. for example, during hate speech annotation, a typical question might ask: "does the above text contain rude, hateful, aggressive, disrespectful, or unreasonable language?" @xcite . previous work has documented that existing datasets gathered through such a process contain mentioned language misclassified as harmful @xcite . this implies that researchers may not typically consider the special case of mentioning statements (including counterspeech) when collecting annotations, or that annotators with varying backgrounds and positionality may not agree on how to treat mentioning statements @xcite . by classifying hate speech and misinformation, nlp tools make implicit judgments regarding when mentioned language is permissible. however, counterspeech should be permissible by definition, as it challenges and opposes harmful narratives @xcite . we suggest that in downstream content classification, treatment of mentioned language should instead be explicitly encoded. more broadly, efforts to use llms for the design of conversational socio-technical systems should take into account the use-mention distinction and strive to embed values regarding how content should be treated. to that end, our prompting mitigation serves as an example of how one societal value can be encoded by leveraging a linguistic construct to specify the treatment of counterspeech. beyond surface features a substantial body of literature has examined shortcomings of online content classification @xcite . our results suggest that many such error cases are linked to the inability to sufficiently distinguish between use and mention. for instance, previous work has suggested that the mere fact that a text contains swear words @xcite , identity terms @xcite related to ethnicity @xcite , religion @xcite , gender, race, and disability @xcite , or dialect markers @xcite increases toxic classifications @xcite . such lexical biases where surface features spuriously influence classification are indicative of inadequate ability to make a use-mention distinction. surprisingly, we find that even the latest models (i.e., gpt-4) are very sensitive to shallow lexical biases. although non-literal language understanding beyond surface features is essential to human communication, large language models tend to struggle with interpretations of subtle utterances @xcite @xcite . our work thus extends prior work aiming to perform more nuanced online content classification @xcite @xcite . future work needs to attend to these subtle and implicit meanings to keep misinformation and hate speech classifiers from being mere surface topic classifiers. beyond counterspeech while we focus on online speech, the use-mention distinction might be important in other contexts like education or tutoring (where mention language occurs frequently to specify spelling or translation), law @xcite , and human-ai interaction @xcite . in general, our work offers promising directions for embedding meta-linguistic reasoning to improve performance on challenging downstream tasks, such as eliciting metaphorical meanings, or dog whistle classifications, where existing methods are not reliable @xcite and teaching metalinguistic skills might offer a possible way forward. ## conclusion this work highlights the theoretical and practical importance of the use-mention distinction in nlp and css. we also provide guidelines and directions for future research in modeling mentioned language and mitigating the impact on downstream tasks where failure to distinguish mention leads to harmful misclassifications.