Text Style Transfer for Bias Mitigation using Masked Language Modeling (2022)

Abstract: It is well known that textual data on the internet and other digital platforms contain significant levels of bias and stereotypes. Various research findings have concluded that biased texts have significant effects on target demographic groups. For instance, masculine-worded job advertisements tend to be less appealing to female applicants. In this paper, we present a text-style transfer model that can be trained on non-parallel data and be used to automatically mitigate bias in textual data. Our style transfer model improves on the limitations of many existing text style transfer techniques such as the loss of content information. Our model solves such issues by combining latent content encoding with explicit keyword replacement. We will show that this technique produces better content preservation whilst maintaining good style transfer accuracy.

## introduction authors such as @xcite and @xcite have drawn attention to some fairness problems in the nlp domain. in a post on buzz-feed @xcite with the title, "scientists taught a robot language. it immediately turned racist", the author reports how various automated language systems are disturbingly learning discriminatory patterns from data. another prominent example of bias in nlp is amazon's ai recruitment tool which turned out to be biased against female applicants @xcite . mitigating bias in textual data before training can be an important preprocessing step in training fair language systems like chatbots, language translation systems, and search engines, but a more direct need for mitigating bias in textual data has been pointed out by various researchers @xcite @xcite who have uncovered the worrying issue of bias in job advertisements. this can have significant implications on the job recruitment process. as a matter of fact, gaucher et al. @xcite explored the effect of biased job advertisements on participants of a survey. they found that changing the wording of a job advertisement to favor a particular gender group considerably reduced the appeal of the job to applicants not belonging to that gender, regardless of the gender stereotype traditionally associated with the job. consequent to such findings, a few tools and models have been developed to detect and mitigate biases in job advertisements. some of these tools include text editors like textio which has been successfully used by companies such as atlassian to increase diversity in their workforce @xcite . another area of impact, regarding biased texts, is in news publications; @xcite explore the issue of hyperpartisan news from an extreme left or right-wing perspective. again, with the prevalence of hate speech and microaggression perpetuated on various social media platforms, there have been growing concerns about fairness in such areas. a machine learning technique that can be employed to mitigate bias in text documents is style transfer. style transfer is a technique that involves converting text or image instances from one domain to another, such that the content and meaning of the instance largely remain the same but the style changes. however, a problem that has challenged research in text style transfer is the relative unavailability of parallel data that would ideally be required to train such models @xcite @xcite . training with parallel data makes it possible to directly map training instances from one domain to the other, hence, facilitating the learning process. due to this, most style transfer systems mainly employ training techniques that fall under two categories: keyword replacement and auto-encoder sequence-to-sequence techniques. in the case of keyword replacement, biased words are deleted and replaced with alternative words. in the case of the auto-encoder sequence-to-sequence generative approach, the input text is directly encoded by an encoder to get a latent representation of the text, which is subsequently decoded by a decoder. the main contributions of this work include: we make the code and data used in this work available foot_1 . ## style transfer style transfer has been widely explored in computer vision to convert images from one style to another @xcite @xcite . however, directly applying image style transfer techniques for text is problematic because of the unique characteristics of both domains. for instance, in text, style and content are more tightly coupled and harder to separate @xcite . in addition to that, the non-differentiability of discrete words causes optimization problems @xcite . in nlp, style transfer has mostly been explored in areas such as sentiment analysis @xcite @xcite and machine translation @xcite . a few style transfer learning techniques use parallel data for training. @xcite give an elaborate survey on such models. in this paper, we will only focus on models that are trained on non-parallel data, some of which we will review in the following subsection. ## methodology the goal of our model is to transform any piece of biased text into a neutral version. if we take the two style attributes s a and s b to represent neutral style and biased style respectively, given a text sample x b that belongs to s b , our goal is to convert x b to x a , such that x a belongs to style s a but has the same semantic content as x b except for style information. our model is composed of four main components, as illustrated in fig 1 . we also illustrate the process with an example in fig. 2 . ## experiments for our experiments, we focus on gender bias (we limit our work to a binary definition of gender) foot_2 . the use of gender is motivated by the relative availability of resources such as datasets. nonetheless, we believe that our work is adaptable to other forms of biases such as racial bias since the technique is not dependent on the domain (only neutral and bias examples are needed). to show our technique's applicability in different domains, we experiment on gender obfuscation, where instead of mitigating the bias, we try to convert female-authored texts to "look like" male-authored texts. we arbitrarily chose to convert from female to male just for the sake of experiment; the same technique can be applied for male to female as well. all experiments are conducted using english language corpus. in the future, we hope to extend our work to cover other languages as well. we discuss the details of our experiments in the following subsections. ## conclusion in this work, we introduce a style transfer model that can be used to mitigate bias in textual data. we show that explicit keyword replacement can be effectively combined with latent content represen-tation to improve the content preservation of text style transfer models. as part of our future work, we intend to expand this work to other languages, we plan to explore possible improvements to the model such as adversarial learning, and also to include human evaluators for qualitative evaluation. again, we intend to investigate other forms of attributes beyond tokens, such as sentence length, and how that affects bias in textual data. we also plan to apply our model as a preprocessing technique to train fair language models. we believe this could significantly reduce biases found in automated language systems.