Content Moderation for Evolving Policies using Binary Question Answering (2023)

Abstract: Content moderation on social media is governed by policies that are intricate and frequently updated with evolving world events. However, automated content moderation systems often restrict easy adaptation to policy changes and are expected to learn policy intricacies from limited amounts of labeled data, which make effective policy compliance challenging. We propose to model content moderation as a binary question answering problem where the questions validate the loosely coupled themes constituting a policy. A decision logic is applied on top to aggregate the theme-specific validations. This way the questions pass theme information to a transformer network as explicit policy prompts, that in turn enables explainability. This setting further allows for faster adaptation to policy updates by leveraging zero-shot capabilities of pre-trained transformers. We showcase improved recall for our proposed method at 95\% precision on two proprietary datasets of social media posts and comments respectively annotated under curated Hate Speech and Commercial Spam policies.

## introduction social media platforms use content moderation to safeguard users from abuse, harassment, malicious attacks, spam, etc. this moderation process is governed by a set of community policies foot_0 . for example, to shield the users from undesired spammy advertising of illegal products/services, social media platforms generally maintain a commercial spam (cs) policy foot_1 . the large volume of content generated on social media platforms necessitates building automated systems for content moderation to scale policy-specific validations. @xcite . traditionally, automated content moderation systems are mostly binary classifiers @xcite often aided by pre-processing @xcite and additional tasks such as intent identification @xcite . recent approaches involve fine-tuned large language models (llms) @xcite , putting attention on suspicious pieces of text @xcite , or reformulating the problem as multi-task learning @xcite or natural language inference @xcite . however, policy compliance in automated content moderation still remains a challenge due to two primary reasons: (1) the governing policies are likely to contain intricacies arising from various aspects like content-specific edge cases, context driven interpretations, and exceptions. for example, the common industry practice considers policy as a single atomic concept and formulates content moderation as binary classification problem. here the policy appears to the classifier as a black-box abstract concept yet it is expected to learn even the minute intricacies of the policy only through the labeled data. this leads to three major production challenges: (1) labeled data are limited in quantity. to get a better understanding of content policies we take an example cs policy that prohibits advertising/selling of illegal products from any of the three categories (hereafter called themes), namely human body parts, recreational drugs, and cryptocurrency (see when we consider the policy as a set of themes combined by a decision logic, it enables us to formulate the task of policy compliance as a binary question answering problem @xcite that leverages a pre-trained large language model (llm) as described in figure 1 . this formulation has four advantages. (1) the theme information can be passed to the llm in the form of explicit prompts, in this case, binary questions (answered yes or no). this is similar to a prompt-based learning @xcite approach that enables a better understanding of the policy. (2) prompting enables leveraging zero-shot capabilities of llms for understanding the question-content relation to validate less prevalent or newly added themes with no or few data samples. (3) the decision logic gets decoupled from the model. this simplifies learning and enables fast adaptation to policy changes. the key highlights of this paper are as follows:. (1) in section 3, we propose a binary question answering based content moderation (qna-cm) system. here, we leverage the policy structure that allows reformulating the problem of content moderation as a generic task of binary qna. going beyond @xcite ; @xcite that deal with more syntactical and factual questions, with qna-cm we aim to answer semantically involved theme-validations. (2) contrary to boolq @xcite in qna-cm to maintain diversity and limit class imbalance in the training set, we undertake a sampling strategy detailed in section 3. (3) we further propose a scalable multi-level inference strategy in section 3 that enables qna-cm to perform at near computational cost of binary classifiers while offering greater explainability. (4) using questions qna-cm leverages explicit policy knowledge and consequently gains agility to policy changes in a zero-shot manner, as demonstrated in a simulation study in section 4. ## related works content moderation systems usually employ binary @xcite or multi-class classifiers @xcite to label content. a binary classifier ignores themes altogether by treating policy as a black box seen through the lens of spam policy theme 1 theme 2 … theme k theme k+1 question 1 question 2 … question k question k+1 llm fine-tuned for binary qna content answer 1 answer 2 … answer k answer k+1 answer k+2 decision logic: combination of answers using and, or, not, etc. updated policy theme 1 theme 2 … theme k theme k+1 theme k+2 question 1 question 2 … question k question k+1 question k+2 spam clear policy updates: • removed theme k+1 and introduces new theme k+2. • the model stops answering for question k+1 and uses zero-shot to answer question k+2. • the decision logic is also updated. ai side business side figure 1 : high-level schematic of the proposed content moderation system. a theme-validating binary answerable question derived from the policy along with a content is passed to llm that answers yes or no. these answers from llm are then aggregated by the well defined decision logic provided by the policy. a policy change at the business side does not impact the ai side as the llm uses zero-shot to answer new questions. and clear labels. even though a multi-class classifier may consider themes it is not agile to policy changes. with the advent of llms @xcite , models like textcnn @xcite , xg-boost (chen and guestrin, 2016) using external word embeddings like glove @xcite have been outperformed. notable llmbased content moderation systems, primarily designed for hate speech detection, usually fail in production due to language-specific interjections @xcite , limited data availability @xcite , demand for fine grained subjective labels @xcite , theme imbalance @xcite , and high response time @xcite . a pre-trained llm can be fine-tuned to perform text understanding tasks such as binary question answering as in boolq @xcite . however, boolq is trained to answer in yes or no responses to content-specific factual questions thus cannot be directly applied to the task of policy compliance. even though boolq inspired question answering along with rule based deductive reasoning have found some success in content validation @xcite , they use simple clearly defined policies and did not investigate the applicability in content moderation that requires answering semantically involved questions. to elaborate, policies governing content moderation often use legal language that are difficult to process by llms @xcite @xcite . for short, focused, and well defined insurance policies, expressing the rules as decision trees may be useful @xcite but that neither extends to capture the intricacies in social media content moderation policies or formally characterize their updates. ## methodology preliminaries: let us take a set @xmath0, x 2 , • • • , x n } of n text contents paired with a set of labels y @xmath1, clear} n where p is the underlying policy. due to the likely imbalance between s spam and c clear contents @xmath2. a policy p as mentioned in section 1 usually consists of a set of themes t and a decision logic (combination of logical operators like and, or, etc.) d to combine the theme-specific validations to reach a final spam or clear label or @xmath3, d). to elaborate, content policies in social media typically have a primary theme of "common intent" in t . for example, in cs this can be "sale of prohibited items" or in hate speech a "hateful sentiment". evidently, if this primary theme is violated then only it makes sense to proceed with the checks on the other themes for finding a spam. each of the other themes individually covers a certain "specific" rule under the policy such as a regulated product like recreational drugs in cs and inherent traits like gender in hate speech. the policy provides the decision tree d that in its commonly preferred form marks a content spam if "common intent" is violated along with (logical and) any (i.e. logical or) other specific theme is contravened. now p when circulated to the user, should phrase t as a guideline to assist the user in creating good quality content. however, when p is provided to human reviewers, t can be rephrased as questions q as that is more intuitive for validating a content. for example, a publicly circulated hate speech policy may state "please do not create hateful contents that target the inherent traits of an individual or group." to a reviewer this may be rephrased as a set of binary answerable questions q along with a decision logic d to aggregate the answers, as shown in figure 2 . similarly, in qna-cm mimicking a human reviewer we rephrase t as q such that the answers of q can be logically combined by d. formally, for a policy with k themes the set @xmath4, q 2 , • • • , q m } contains m validatory questions. here, @xmath5. the decision tree d is a boolean function that maps from {0, 1} m to {0, 1} using logical operators. we are representing yes as 1 and no as 0 to match the implementation while ⊗ and ⊕ respectively denote the logical operators and and or. binary question answering using llms: we follow from llm classifiers @xcite and boolq @xcite , for validating a content @xmath6. the input i concatenates [cls], q, [sep], and x in the order, where [cls] and [sep] are special tokens. the output f (i) of the llm f summarizes the input i in the feature space at f i [cls] i.e. the dimension corresponding to the [cls] input token @xcite . this f i [cls] is sent through fully-connected layers to map to the two classes @xcite . after applying softmax to the logits, the model will output probability scores p r(1|(q, x)) and p r(0|(q, x)) respectively for 1 (yes) and 0 (no) responses. a threshold θ converts the probability scores to binary labels. this can be trained end-to-end with a loss such as binary cross-entropy. figure 4 illustrates the architecture. training of qna-cm: in the qna-cm training set each sample is a question-content pair with 1 (yes) or 0 (no) label. to form such a training data we ask the same set of questions q to every content x in x. however, this may result in severe class imbalance depending on r and m. for simplicity without loss of generality, let us assume @xmath7, a spam content violates only one theme, and a positive theme violation corresponds to a yes answer only. thus, we have a total of sm(r + 1) questions in the training data, where s of them are answered by yes and the rest (s(m -1) from spam and rsm from clear) are answered by no, resulting in an imbalance of (m + mr -1). a naive solution of sampling random no answering question-content pairs may not provide a quality training set. we sample diverse question-content pairs with label 0 (no) in three ways: (s1) pair a clear sample with a random question with probability ν n and assign it a 0 (no) label. (s2) take a spam x that answers yes to q j . pair x with probability ν s with any @xmath8. (s3) use theme-specific weak classifiers using models like textcnn or pre-trained natural language inference models like bart @xcite to find the q j with highest confidence (above ω) that matches with a clear sample x. with a probability ν h , pair x with q j and label it as 0 (no). the "common intent" of a policy is expressed through the top-level spam and clear labels. to utilize this additional information and learn the commonalities across themes to aid zero-shot generalization, we sample spam and clear contents respectively with probability ν + and ν -and pair them with a "common intent" validating question in the training set. note that, in the process of building the training set, we introduce five new data dependent hyperparameters in qna-cm, namely ν s , ν h , ν n , ν + , ν -, and ω. inference using qna-cm: we propose a scalable multi-level inference strategy for qna-cm as described in figure 3 . here we exploit two peculiarities of the content moderation ecosystem. (1) spammy content is commonly very less frequent than clear content. (2) in our two-level inference strategy. in the first level l 1 , we match the content against a single question representing the common intent of the policy. only if the content is matched in l 1 , we proceed to the second level of l 2 to validate it against all the m theme-specific questions, otherwise we directly mark it as clear. this way only the potential spammy content will be validated against all m theme-validating questions while the rest will be cleared in l 1 with a similar computation cost of a binary classifier. in other words, the computational overhead will only be zm/(1 + r) times in practice where z is the ratio of potential spam to actual spam content. typically in production @xmath9, thus asserting hate speech content q d does this content mention race? does this content mention ethnicity, national origin, religious affiliation, or immigration status? does this content mention sex or gender identity? does this content mention disability? does this content mention sexual orientation? does this content mention caste? inherent traits does this content mention hateful speech? sentiment figure 2: set of questions q (derived from t ) and decision logic d for a sample hate speech policy. want to gain weight and have muscled body in a month? try our miraculous product now! does this content mention prescription drugs or supplements? does this content mention cannabis and its derivatives? llm with fully-connected layer score: 0.02 score: 0.90 content theme validating questions commercial spam d llm with fully-connected layer on top does this content promoting, advertising, or attempting to sell? common intent of the policy validating question clear answer: no score: 0.98 answer: yes clear answers: no l 1 l 2 discarded theme inserted theme inserted theme score figure 3 : the multi-level inference of qna-cm is illustrated with the cs policy example. the thresholds for both levels are set at 0.5. in l 1 the content gets a high score of 0.98 for the common policy intent to move forward to l 2 . during l 2 we pair the content with each of the m theme validating questions and get the scores. here, given the content violates pharmaceuticals theme it gets a high score of 0.9 for the same and obtains low scores otherwise. thus, the content gets yes for pharmaceuticals and consequently gets labeled as spam. ## experiments experimental protocol: we use two proprietary long-tailed text datasets, both sampled from content publicly posted on social media during 2021-2023 (both are sampled from the same social media to maintain consistency). we take 4 english and 2 multilingual pre-trained llms for our experiments (listed in appendix a.4 with additional network architecture and hyperparamater details). for all the experiments we compare a fine-tuned llm-based baseline binary classifier that maps the [cls] token embedding of the backbone to spam or clear labels by a multi-layer perceptron network againstst a qna-cm model with the same backbone. content moderation systems in production aim to achieve better recall with high precision such that the users (and reviewers) are minimally affected by false positives. thus, we use recall value at 95% precision level for comparing among contenders (note that accuracy is not suitable given the class imbalance while gmean is not informative as content moderators do not need to focus on true negatives @xcite ). we demonstrate performance of qna-cm across two dimensions. (1) to train with the full dataset, we retain all the themes in training, validation, and test set mimicking a long standing static policy. here we aim to evaluate how well qna-cm employs prompting through questions to handle the long tallied theme distribution. (2) for the cs policy, we simulate a policy change (hate speech policy is commonly static as the inherent traits are well defined) where new themes are introduced in the policy (see appendix a.5). thus, we train the model only on previously existing themes while inferring on newly introduced ones to evaluate how qna-cm employs prompting and zero-shot capability of llm to adapt to policy update. 12 ) that for both the policies, qna-cm is achieving a better performance on average over the four backbone llms than the baseline binary classifiers in terms of r@95p. the performance improvement is more apparent in the case of simulated policy changes in cs, indicating better adaptability of qna-cm to policy updates. further, in the training with full data, the better performance of qna-cm attests to the usefulness of theme-specific knowledge prompted through the questions. explainability of qna-cm: in importance of prompted learning: we showcase benefits of our prompt based learning framework qna-cm on hate speech detection. in the real world industry setting, hate speech detection is often plagued with false positives that arise due to binary classifiers getting confused between "hateful" content and "hate speech". this distinction is important for fair assessment of the content severity and user regulation. therefore, understanding of inherent traits along with hateful intent becomes crucial, which can be achieved via questions in qna-cm. we demonstrate this setting by two experiments. (1) at 95% precision level we compare the average theme-specific recall over four backbone llms for each of the 6 inherent traits for the d2 posts dataset labeled against the hate speech policy. from ## conclusion in this work, we model content moderation as a binary question answering problem where the questions act like prompts, validating various themes belonging to a content policy. this further allows faster adaptation to policy updates by leveraging zero-shot capabilities of pre-trained transformers. our experiments show 7% absolute improvement in recall over the binary classification setting. in case of policy updates we achieve 22% absolute recall improvement as well, without any additional training. furthermore we show improved recall on multilingual data with qna-cm fine-tuned only on english. we also show improved recall and reduced false positives for hate speech using qna-cm. all these facilitate an agile response to policy updates by prompt injection thus limiting member exposure to spam. in the future, we aim to investigate the applicability of open source datasets with recently developed large generative models with high natural language understanding and promptdriven zero-shot capabilities such as gpt series @xcite or llama @xcite in the qna-cm framework. facebook hate speech policy ( https: //transparency.fb.com/en-gb/policies/ community-standards/hate-speech/), and twitter policy on hateful conduct ( https:// help.twitter.com/en/rules-and-policies/ hateful-conduct-policy). however, like the cs policy the hate speech policy curated by us does not directly follow any of the social media in particular rather it combines the essence of the three, focussing on some key inherent traits only. moreover, we have observed that even highly capable llms like bart @xcite often get confused with particularly identifying fine grained inherent traits. for example, it is a complicated problem for the llms to disambiguate between ethnicity, national origin, religious affiliation, and immigration status as they often occur together and can even be interpreted synonymously in the same content. similarly it has been found that llms are not powerful enough to properly distinguish between gender and sex as different inherent traits. hence, while creating the question set as described in figure 2 we joined these fine grained inherent traits together. one key challenge for qna-cm is to formulate questions from the policy definition as the same statement can be rephrased in multiple ways. to remove this possible ambiguity during training we left the question design to the unanimous decision by a team of 5 reviewers who are well acquainted with the two curated policies. we felt this is a reasonable approach as this directly reflects the human reviewers' understanding of a content policy and best aids the mimicking of that in qna-cm.