BERT -Beta: A Proactive Probabilistic Approach to Text Moderation (2021)

Abstract: Text moderation for user generated content, which helps to promote healthy interaction among users, has been widely studied and many machine learning models have been proposed. In this work, we explore an alternative perspective by augmenting reactive reviews with proactive forecasting. Specifically, we propose a new concept text toxicity propensity to characterize the extent to which a text tends to attract toxic comments. Beta regression is then introduced to do the probabilistic modeling, which is demonstrated to function well in comprehensive experiments. We also propose an explanation method to communicate the model decision clearly. Both propensity scoring and interpretation benefit text moderation in a novel manner. Finally, the proposed scaling mechanism for the linear model offers useful insights beyond this work.

## introduction text moderation is essential for maintaining a nontoxic online community for media platforms @xcite . many efforts from both academia and industry have been made to address this critical problem. recently, the most prototypical thread is to do sophisticated feature engineering or develop powerful learning algorithms @xcite @xcite @xcite . automatic comment moderation schemes plus human review are certainly the cornerstone of the fight against toxicity. these existing works, however, are reactive approaches to handling user generated text in response to the publication of new articles. in this paper, we revisit this challenge from a proactive perspective. specifically, we introduce a novel concept text toxicity propensity to quantify how likely an article is prone to incur toxic comments. this is a proactive outlook index for news articles prior to the publication, which differs radically from the existing reactive approaches to comments. in this context, reactive describes comment-level moderation algorithms after the publication of news articles (e.g., perspective @xcite ), which quantifies whether comments are toxic and should be taken down or sent for human review. proactive emphasizes article-level moderation effort before the publication (without access to comments), which forecasts how likely articles are to attract toxic comments in the future and gives suggestions (e.g., rephrase news articles properly) in advance. our work can be viewed as the first machine learning effort for a proactive stance against toxicity. formally, we propose a probabilistic approach based on beta distribution (beta) to regress article toxicity propensity on article text. for previously published news articles with comments, we take the average of comments' toxicity scores as the ground-truth label for model learning. the effectiveness of this approach is shown in both test set and human labeling. we also develop a scheme that can provide convincing explanation to the decision of the deep learning model. ## related works to our best knowledge, there's no prior research thread on proactive text moderation. nonetheless, many reactive approaches have been explored including hand-crafted feature engineering @xcite @xcite , neural networks @xcite @xcite and transformer variants @xcite . recently, context, in the form of parent posts, has been studied but it is only viewed as regular text snippets for lifting the performance of toxicity classifiers @xcite while screening posts. our work instead focuses on predicting the proactive toxicity propensity of articles before they receive user comments. beta distribution is usually utilized as a priori in bayesian statistics. the most popular example in natural language processing is topic model @xcite , where the multivariate version of beta distribution (a.k.a. dirichlet distribution) generates parameters of mixture models. beta regression is originally proposed for modeling rate and proportion data @xcite ) by parameterizing mean and dispersion and regressing parameters of interest. it has been applied to evaluate grid search parameters in optimization (mckinney-bock and bedrick, 2019), model emotional dimensions @xcite and statistical processes of child-adult linguistic coordination and alignment @xcite . ## beta regression in this work, both comment toxicity score and the derived article toxicity propensity score (to be detailed in the subsequent section 4.1) range from 0 to 1. empirically, their distributions exhibit an asymmetry and may not be modelled well with the gaussian distribution (figs. 2 and 3 of appendix a). furthermore, comment toxicity score distributions of individual articles vary with article content as shown in fig. 3 of appendix a. modelling the entire distribution of an article comment toxicity scores is thus a reasonable approach. beta distribution is very flexible and it can model quite a wide range of well-known distribution families from symmetric uniform (α = β = 1) and bellshaped distributions (α = β = 2) to asymmetric shapes (α = β). in this context, the toxicity propensity score y is assumed to follow the beta distribution with probability density function (pdf): substituting eqs. 1 and 2 into it gives the final objective function. in the inference phase, with learned f α (•), f β (•) and g(•), α m and β m for a new sample x m can be readily derived from eq. 2. we take the mean of eq. 1 as a point estimator: y @xmath0. ## additional study linear regression (bow-mse) is inferior to bertβ. nonetheless, it is much faster in training, inference and explanation as it is about 20 times as small as bert-β. thus, we investigate if the performance of the linear model could be improved for industrial deployment. inspired by nbsvm (wang and manning, 2012), we scale tf-idf vectors of bow-mse by a weight vector defined as @xmath1, as compared in our work can benefit text moderation. the proactive propensity offers a toxicity outlook for comments, which could be utilized in multiple ways. for example, stricter moderation rules are enforced for articles that are predicted to have a high toxicity propensity. furthermore, the propensity could be used as an additional feature for the downstream reactive toxicity recognition models, as well as for allocation of appropriate human resources. the explanation tool can also be used to remind editors to rephrase some controversial words to mitigate the odds of attracting toxic comments. text moderation is an important yet challenging task, our proactive work is attempting to open up a new perspective to augment the traditional reactive procedure. our current model, however, is not perfect as shown by article b in fig. 3 of appendix a where the learned distribution doesn't fit well the observed histogram. technically, nblr is an encouraging lightweight extension to linear regression. likewise, we will continue to work towards the improvement of the non-linear beta regression. ## conclusion we approach text moderation by developing a wellmotivated probabilistic model to learn a proactive toxicity propensity. an explanation scheme is also proposed to visually explain the connection between this new prospective score, and text content. our experiment shows the superior performance of the proposed bert-β algorithm, compared with a number of baselines, in predicting both the average toxicity score, and the human judgement.