Performance and Risk Trade-offs for Multi-word Text Prediction at Scale (2023)

Abstract: Large Language Models such as GPT-3 are well-suited for text prediction tasks, which can help and delight users during text composition. LLMs are known to generate ethically inappropriate predictions even for seemingly innocuous contexts. Toxicity detection followed by filtering is a common strategy for mitigating the harm from such predictions. However, as we shall argue in this paper, in the context of text prediction, it is not sufficient to detect and filter toxic content. One also needs to ensure factual correctness and group-level fairness of the predictions; failing to do so can make the system ineffective and nonsensical at best, and unfair and detrimental to the users at worst. We discuss the gaps and challenges of toxicity detection approaches - from blocklist-based approaches to sophisticated state-of-the-art neural classifiers - by evaluating them on the text prediction task for English against a manually crafted CheckList of harms targeted at different groups and different levels of severity.

## introduction large language models (llms) are powerful, yet known to generate potentially risky, harmful, offensive texts @xcite , even when the context is seemingly innocuous @xcite . while there are several studies that propose techniques for measurement and mitigation of biases of llms @xcite @xcite @xcite , there are very few that analyze such harms in context of real-world downstream applications. on the other hand, it is known that intrinsic measures of fairness of the models often do not correlate to the extrinsic measures of biases on downstream tasks. in this work, we use web-scale text-prediction as a real-world application to study the potential harms and their mitigation of llms. text predic-tion is a commonly used writing assistance technology which provides real-time, context-dependent word and phrase completions as a user is typing @xcite @xcite . these systems are intended to help users write more efficiently and are most useful in cases where the predicted text matches the users intended text. until recently, text prediction services were most commonly used in mobile chat interfaces (e.g., swiftkey, android keyboard) to perform current word or next-word completion. due to the rise of llms, text prediction services now provide accurate, multi-word predictions across a variety of interfaces and applications, such as email @xcite and document composition. text prediction can affect the process and the product of writing @xcite therefore, it can influence users to write harmful content. word and phrase-based blocklists provide a simple yet effective mitigation solution and are widely used in commercial systems today @xcite . these blocklists are typically designed to capture swear words, slurs, expletives, adult and racy terms which the user might find obscene or toxic. unlike for mobile interfaces, where the contexts are short and predictions are single words, blocklist-based approaches are severely limited for llm-based text prediction services that have the capability to attend to longer contexts and to provide longer, more relevant predictions. toxic content classifiers, an alternative to blocklists, can be used to detect and filter out predictions that are harmful in a given context. prior research has provided publicly available classifiers @xcite @xcite , but as we will demonstrate in this work, such classifiers are not suitable for mitigating harms in text prediction services. this is primarily due to the following two reasons: first, in many cases the context (text typed by the user so far) is innocuous; however, any prediction might make it toxic or potentially harmful when considering the full context with the prediction in a way that the current toxicity classifiers do not consider toxic. e.g., you don't look like a mexican second, certain factually incorrect predictions might also be interpreted as potentially harmful or polarizing, e.g., ## experimental setup keeping in mind our objective to measure the effectiveness of the various harm mitigation strategies, all our experiments are designed around the same text-predictor which does not contain any explicit harm filtering technique. this will serve as our baseline. we then apply toxic content filtering techniques at two levels. first, at the level of the contextc i-(k-1)...i . we shall call this the pre-filter. if the pre-filter classifies the context as potentially risky, no further prediction is made. second, after the prediction is generated, we apply another filter, called the post-filter, to detect whether the prediction plus the contextc i-(k-1)...i • c i 1...k i -is potentially harmful. if so, the prediction is dropped again. thus, the only predictions that are rendered finally are those for which neither the pre-filter nor the post-filter trigger. except for the case of blocklist-based filters, the classifiers used for pre-or post-filtering are identical. also, for the ease of comparison and to avoid combinatorial explosion, in all setups we shall use the same classifier as the pre-and post-filter instead of mixing them. ## checklist of harms checklist @xcite is a behavioral testing approach for nlp systems, in which unit tests are generated from templates capturing capabilities that the system must possess. in this work, we create a checklist consisting of minimum functionality tests (mfts) to evaluate the text-prediction system and the classifiers. ## conclusion the current study highlights three important aspects of the text prediction task. first, it is difficult to estimate the risks of a text predictor due to unavailability of appropriate datasets. second, off-theshelf toxicity classifiers have higher leakage ratios than what is acceptable. although blocklists provide a potential solution, their context-insensitive nature makes them an extremely conservative solution for long form text prediction. third, llm based text-predictors are inherently biased towards more toxic/no/innocuous predictions towards certain groups, and while classifiers can improve the fairness of prediction across the groups, this comes at a cost of suppressing most predictions and bringing down the overall usefulness of the system. thus, responsible text-prediction at scale offers several research challenges involving complex trade-off between performance on one hand and risks and fairness on the other. please contact the last author for the checklists and their fine grained annotations created during this work.