Dynamic Forecasting of Conversation Derailment (2021)

Abstract: Online conversations can sometimes take a turn for the worse, either due to systematic cultural differences, accidental misunderstandings, or mere malice. Automatically forecasting derailment in public online conversations provides an opportunity to take early action to moderate it. Previous work in this space is limited, and we extend it in several ways. We apply a pretrained language encoder to the task, which outperforms earlier approaches. We further experiment with shifting the training paradigm for the task from a static to a dynamic one to increase the forecast horizon. This approach shows mixed results: in a high-quality data setting, a longer average forecast horizon can be achieved at the cost of a small drop in F1; in a low-quality data setting, however, dynamic training propagates the noise and is highly detrimental to performance.

## introduction the flow of a conversation can be interrupted when its participants turn to attacking each other as opposed to communicating constructively. the task of forecasting derailment aims to predict outbursts of hostile behavior between users in online conversations before they happen @xcite . this stands in opposition to related tasks like hate speech detection @xcite , where abusive language is identified post factum. accurate forecasts could give forum moderators an early warning of troublesome exchanges or even serve as a step towards the prevention, human-driven or automatic, of derailment. as a personal attack can occur at any point during an unfolding conversation, forecasting has to happen dynamically. furthermore, the earlier the warning for a potential derailment comes, the more actionable it is. in the example from the first comment made by user a, and hope to get an early warning for the derailment coming up in two turns. this is indeed the inference procedure that was used in previous work on the task @xcite . training, on the other hand, was done statically in those works, i.e. by feeding all turns up to the personal attack as input and making a prediction based on that. this leads to a disparity between the training and inference procedures, where at training time the implicit task of the model is to predict whether the next turn contains a personal attack, whereas at inference time its task is to predict whether any future turn will contain a personal attack. we propose to bridge this gap by training the model dynamically instead of statically. we evaluate our dynamic approach on the two available data sets for forecasting derailment, conversations gone awry, built from gold-standard annotations for the task, and reddit changemyview, built from partial annotations derived from moderator actions. our experiments reveal that dynamic training does lead to earlier forecasts, but not without a loss in accuracy and f1. in the highquality data setting the loss is small (0.5% f1), but in the lower-quality data setting, dynamic training appears to multiply any noise present in the data and is therefore detrimental to model performance. a final, somewhat trivial, but necessary contribution of our work is the application of a pre-trained transformer language encoder to the task, which sets a high new baseline, improving over previous results on the two data sets by 2.1 and 1.2 points on the f1 metric, respectively. ## resources for forecasting derailment in this section, we describe the two main resources available for the task of forecasting derailment. 1 conversations gone awry (cga) cga @xcite was built from wikipedia talk page conversations. the initial seed of conversations is sampled from wikiconv based on an automatic measure of toxicity that ranges from 0 (no toxicity) to 1 @xcite . 2 a conversation is included as a potential example of derailment if the n th comment in it has a toxicity score higher than 0.6 and all preceding comments have a score lower than 0.4. a conversation is included as a potential example of non-derailment if all comments in it score below 0.4. this initial seed is subsequently manually annotated (1) to confirm that the initial exchange is civil and (2) to determine whether after this initial exchange a personal attack occurs from one user towards another. conversations with a personal attack are included in the dataset up to and including the turn containing the personal attack. this means that in effect all n -1 turns in a conversation are civil and the n th one is optionally civil or contains a personal attack. topic control is further implemented by pairing each derailed conversation with a non-derailed conversation from the same talk page. this procedure yields a dataset of 4,188 conversations, split 60-20-20, with a balanced representation of the two classes. reddit changemyview (cmv) @xcite built cmv from reddit conversations held under the change-myview subreddit, using as partial annotations the deletion of a turn by the platform moderators when that was done under reddit's rule 2: "don't be rude or hostile to other users." topic control and class balancing are implemented here, too. yet, there is no control to ensure that all turns in a conversation prior to the last one would be civil, which as the authors point out is a potential source of noise in the data. the resulting dataset consists of 6,842 data points, with a split of 60-20-20. ## methods for forecasting derailment in this section we describe previous approaches to the task of forecasting derailment in online conver-sations, as well as the new approach we propose. ## results and discussion in in a comparison of values 3 and 5 for k we found the former to give a better f1 score (the latter triggering a high rate of false positives). 5 we further found the combination of α = 1.5 and β = 0.5 to work better than α = 1.0, β = 0.5. the takeaway here is that dynamic training has to be fairly conservative, presumably because in many cases the eventual derailment of a conversation presents with no early sign. as we see in a much larger drop is seen on these metrics for the cmv dataset, however, rendering the approach unfit for this context. ## conclusion and final remarks dynamic training has the desired effect of extending the forecast horizon, at a justifiable cost for high-quality data, but not for noisy data. the horizon of a forecast is important, but a more sophisticated approach may be necessary to achieve a long horizon when data is noisy, e.g. one that considers the exact structure of conversation derailment. the analysis of the composition of cga and cmv also raises an issue with the definition of the task: it is perhaps better to view the forecasting of toxic personal attacks as a subtask within a more general task of forecasting derailment in conversationwith other types of derailment still remaining unexplored. in recent years, a fine-grained typology of antisocial behaviors has been developed @xcite , under which derailments in conversation is seen as the result of various types of antisocial behavior beyond just personal attacks.