Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company’s Reputation (2021)

Abstract: Not all topics are equally “flammable” in terms of toxicity: a calm discussion of turtles or fishing less often fuels inappropriate toxic dialogues than a discussion of politics or sexual minorities. We define a set of sensitive topics that can yield inappropriate and toxic messages and describe the methodology of collecting and labelling a dataset for appropriateness. While toxicity in user-generated data is well-studied, we aim at defining a more fine-grained notion of inappropriateness. The core of inappropriateness is that it can harm the reputation of a speaker. This is different from toxicity in two respects: (i) inappropriateness is topic-related, and (ii) inappropriate message is not toxic but still unacceptable. We collect and release two datasets for Russian: a topic-labelled dataset and an appropriateness-labelled dataset. We also release pre-trained classification models trained on this data.

## introduction the classification and prevention of toxicity (malicious behaviour) among users is an important problem for many internet platforms. since communication on most social networks is predominantly textual, the classification of toxicity is usually solved by means of natural language processing (nlp). this problem is even more important for developers of chatbots trained on a large number of usergenerated (and potentially toxic) texts. there is a well-known case of microsoft tay chatbot 1 which was shut down because it started producing racist, sexist, and other offensive tweets after having been fine-tuned on user data for a day. however, there exists a similar and equally important problem, which is nevertheless overlooked by the research community. this is a problem of texts which are not offensive as such but can express inappropriate views. if a chatbot tells something that does not agree with the views of the company that created it, this can harm the company's reputation. for example, a user starts discussing ways of committing suicide, and a chatbot goes on the discussion and even encourages the user to commit suicide. the same also applies to a wide range of sensitive topics, such as politics, religion, nationality, drugs, gambling, etc. ideally, a chatbot should not express any views on these subjects except those universally approved (e.g. that drugs are not good for your health). on the other hand, merely avoiding a conversation on any of those topics can be a bad strategy. an example of such unfortunate avoidance that caused even more reputation loss was also demonstrated by microsoft, this time by its chatbot zo, a tay successor. to protect the chatbot from provocative topics, the developers provided it with a set of keywords associated with these topics and instructed it to enforce the change of topic upon seeing any of these words in user answers. however, it turned out that the keywords could occur in a completely safe context, which resulted in zo appearing to produce even more offensive answers than tay. 2 therefore, simple methods cannot eliminate such errors. thus, our goal is to make a system that can predict if an answer of a chatbot is inappropriate in any way. this includes toxicity, but also any answers which can express undesirable views and approve or prompt user towards harmful or illegal actions. to the best of our knowledge, this problem has not been considered before. we formalize it and present a dataset labeled for the presence of such inappropriate content. even though we aim at a quite specific taskdetection of inappropriate statements in the output of a chatbot to prevent the reputational harm of a company, in principle, the datasets could be used in other use-cases e.g. for flagging inappropriate frustrating discussion in social media. it is also important to discuss the ethical aspect of this work. while it can be considered as another step towards censorship on the internet, we suggest that it has many use-cases which serve the common good and do not limit free speech. such applications are parental control or sustaining of respectful tone in conversations online, inter alia. we would like to emphasize that our definition of sensitive topics does not imply that any conversation concerning them need to be banned. sensitive topics are just topics that should be considered with extra care and tend to often flame/catalyze toxicity. the contributions of our work are three-fold: we open the access to the produced datasets, code, and pre-trained models for the research use. foot_1 ## related work there exist a large number of english textual corpora labeled for the presence or absence of toxicity; some resources indicate the degree of toxicity and its topic. however, the definition of the term "toxicity" itself is not agreed among the research community, so each research deals with different texts. some works refer to any unwanted behaviour as toxicity and do not make any further separation @xcite . however, the majority of researchers use more fine-grained labeling. the wikipedia toxic comment datasets by jigsaw @xcite @xcite are the largest english toxicity datasets available to date operate with multiple types of toxicity (toxic, obscene, threat, insult, identity hate, etc). toxicity differs across multiple axes. some works concentrate solely on major offence (hate speech) @xcite , others research more subtle assaults @xcite . offenses can be directed towards an individual, a group, or undirected @xcite , explicit or implicit @xcite . insults do not necessarily have a topic, but there certainly exist toxic topics, such as sexism, racism, xenophobia. waseem and hovy (2016 such a topic-based view of toxicity causes unintended bias in toxicity detection -a false association of toxicity with a particular topic (lgbt, islam, feminism, etc.) @xcite . this is in line with our work since we also acknowledge that there exist acceptable and unacceptable messages within toxicity-provoking topics. the existing work suggests algorithmic ways for debiasing the trained models: xia et al. (2020) train their model to detect two objectives: toxicity and presence of the toxicity-provoking topic, @xcite al. (2020) the main drawback of topic-based toxicity detection in the existing research is the ad-hoc choice of topics: the authors select a small number of popular topics manually or based on the topics which emerge in the data often, as @xcite . @xcite suggest a taxonomy of harmful online behaviour. it contains toxic topics, but they are mixed with other parameters of toxicity (e.g. direction or severity). the work by @xcite is the only example of an extensive list of toxicity-provoking topics. this is similar to sensitive topics we deal with. however, our definition is broader -sensitive topics are not only topics that attract toxicity, but they can also create unwanted dialogues of multiple types (e.g. incitement to law violation or to cause harm to oneself or others). ## inappropriateness and sensitive topics consider the following conversation with an unmoderated chatbot: the notion of inappropriateness of a text in our setting is tightly related to this text's topic. this is different from the notion of toxicity which does not have to be topic-dependent. toxic texts are undoubtedly inappropriate in the sense that they should not appear in a respectful conversation. still, they have been actively researched, so we do not consider them in our work. ## topic labeling our final goal is to label the data with inappropriateness, and the sensitive topics are not a goal per se, but mainly a way to define inappropriateness. therefore, we use sensitive topics as a way of data pre-selection. analogously to toxicity, inappropriateness does not often occur in randomly picked texts, so if we label all the messages we retrieve, the percentage of inappropriate utterances among them will be low. thus, our labeling process includes three stages: (i) we collect the dataset of sentences on sensitive topics, (ii) we build a classifier of sensitive topics on this dataset, (iii) we collect the texts on sensitive topics using the classifier and then label them as appropriate or inappropriate. ## appropriateness labeling we should again emphasize that not every utterance concerning a sensitive topic should be banned. while the topic of a text can be sensitive, the text itself can nevertheless be appropriate. thus, we collect the texts on sensitive topics and then label them as appropriate or inappropriate. our initial plan was to define the appropriate and inappropriate subtopics for each topic. however, determining the appropriateness criteria explicitly turned out to be infeasible. therefore, we rely on the inherent human intuition of appropriateness. we provide annotators with the following context: a chatbot created by a company produces a given phrase. we ask to indicate if this phrase can harm the reputation of the company. we also reinforce the annotators' understanding of appropriateness with the training examples. as in the topic labeling setup, here we ask the workers to complete the training before labeling the data. we also fine-tune their understanding of appropriateness with extra training during labeling. analogously to topic labeling, the appropriateness labeling is performed via yandex.toloka crowdsourcing platform. an example of the task interface is given in figure 2 . our crowdsourcing setup repeats the one we used in the topic labeling project. we perform training and quality control analogously to topic labeling. although the appropriateness is not explicitly defined, the workers demonstrate a good understanding of it. their average performance on the training and control tasks is around 75-80%, which indicates high agreement. the average labeling confidence computed via the dawid-skene method is 0.956. the primary sources of the samples passed to appropriateness labeling are the same as in the topic labeling setup (2ch.hk and otvet.mail.ru websites). before handing texts to workers, we filter them as described in section 4.1 and also perform extra filtering. we filter out all messages containing obscene language and explicit toxicity. we identify toxicity with a bert-based classifier for toxicity detection. we fine-tune rubert model (kuratov and arkhipov, 2019) on a concatenation of two russian language toxic comments datasets released on kaggle @xcite .we filter out sentences which were classified as toxic with the confidence greater than 0.75. as mentioned above, toxicity is beyond the scope of our work, because it has been researched before. therefore, we make sure that messages which can be automatically recognized as toxic are not included in this dataset. inappropriate messages in our formulation concern one of the sensitive topics. therefore, we pre-select data for labeling by automatically classifying them with sensitive topics. we select the data for labeling in the following proportion: the further labeling process is performed analogously to topic labeling. we use the same training and quality control procedures and define the number of workers per example dynamically. to get the final answer, we use the same dawid-skene aggregation method. it aggregates the labels given by workers (0 and 1, which state for "appropriate" and "inappropriate") into a single score from 0 to 1. we interpret this score as the appropriateness level, where the score in the interval [0, 0.2] indicates appropriate sentences, the score in [0.8, 1] means that the sentence is inappropriate, and other scores indicate ambiguous examples. ## datasets statistics we collected two datasets: (i) the dataset of sensitive topics and (ii) the appropriateness dataset. the dataset of sensitive topics consists of 25,679 unique samples. 9,946 samples were labeled with a crowdsourcing platform, nearly 1,500 samples were labeled by our team and the rest samples were collected by using keywords from specialized sources. the average confidence of the crowdsourcing annotation is 0.995; the average number of annotations per example is 4.3; the average time to label one example is 10.8 seconds. the appropriateness dataset consists of 82,063 unique samples. 8,687 of these samples also belong to the sensitive topics dataset and thus have manually assigned topic labels. the other 73,376 samples have topic labels defined automatically using a bert-based topic classification model (described in section 7). the average confidence of the annotation is 0.956; the average number of annotations per example is 3.5; the average time to label one example is 7 seconds. 0 1 2 3 4+ topics per sample 0 2500 5000 7500 10000 12500 15000 17500 # samples 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 percentage figure 3: distribution of samples by number of topics. one sample can relate to more than one topic. our analysis showed 15% of such examples in the data (see figure 3 ). the co-occurrence of topics is not random. it indicates the intersection of multiple topics. the most common co-occurrences are "politics, racism, social injustice", "prostitution, pornography", "sex minorities, pornography". in contrast, 13% of samples in the topic dataset do not touch any sensitive topic. these are examples that were pre-selected for manual topic labeling using keywords and then were labeled as not related to the topics of interest. they were added to the dataset so that the classifier trained on this data does not rely solely on keywords. the samples in the datasets are mostly single sentences; their average length is 15 words for the appropriateness dataset and 18 words for the topic dataset. the sample length for different topics ranges from 14 to 21 words. we noticed a strong correlation (spearman's r of 0.72) between the number of samples of a particular topic in the data and the average number of words per sample for this topic. we cannot define if this is a spurious correlation or topics that feature longer sentences tend to be better represented in the data. longer sentences might be easier to annotate. ## evaluation we confirm the usefulness of the collected data by training classification models on both datasets. we fine-tune pre-trained rubert model (bert trained on russian texts (kuratov and arkhipov, 2019)) on our data. we use the implementation of bert-classifier from deeppavlov 11 library with pre-trained conversational rubert weights. 12 ## conclusions and future work we introduce the task of detecting inappropriate utterances -utterances that can cause frustration or harm the reputation of a speaker in any way. we define the notion of a sensitive topic tightly related to the notion of appropriateness. we collect two datasets for the russian language using a large-scale crowdsourcing study. one is labeled with sensitive topics and another with binary appropriateness labeling. we show that while being fine-grained notions, both inappropriateness and sensitivity of the topic can be detected automatically using neural models. baseline models trained on the new datasets are presented and released. a promising direction of future work is improving the performance of the presented baselines, e.g. by using the topic and appropriateness labeling jointly, switching to other model architectures, or ensembling multiple models. another prominent direction of future work is to transfer the notion of appropriateness to other languages by fine-tuning cross-lingual models on the collected datasets.