Generating Intuitive Fairness Speciﬁcations for Natural Language Processing
Abstract
Text classiﬁers have promising applications in high-stake tasks such as resume screening and content moderation.
These classiﬁers must be fair and avoid discriminatory decisions by being invariant to perturbations of sensitive attributes such as gender or ethnicity.
However, there is a gap between human intuition about these perturbations and the formal similarity speciﬁcations capturing them.
While existing research has started to address this gap, current methods are based on hardcoded word replacements, resulting in speciﬁcations with limited expressivity or ones that fail to fully align with human intuition (e.g., in cases of asymmetric counterfactuals).
This work proposes novel methods for bridging this gap by discovering expressive and intuitive individual fairness speciﬁcations.
We show how to leverage unsupervised style transfer and GPT-3’s zero-shot capabilities to automatically generate expressive candidate pairs of semantically similar sentences that differ along sensitive attributes.
We then validate the generated pairs via an extensive crowdsourcing study, which conﬁrms that a lot of these pairs align with human intuition about fairness in toxicity classiﬁcation.
We also show how limited amounts of human feedback can be leveraged to learn a similarity speciﬁcation.
Introduction
Text classiﬁers are being employed in tasks related to automated hiring [1], content moderation [2] and reducing the toxicity of language models [3].
However, they were shown to exhibit biases based on sensitive attributes, e.g., gender [4] or demographics [5], even for tasks in which these dimensions should be irrelevant.
This can lead to unfair decisions, distort analyses based on these classiﬁers or propagate undesirable stereotypes to downstream applications.
The intuition that certain demographic indicators should not inﬂuence decisions can be formalized in terms of individual fairness [6], which posits that similar inputs should be treated similarly .
In a classiﬁcation setting we assume similar treatment for two inputs to require both inputs to be classiﬁed the same, while the notion of input similarity captures the intuition that certain input characteristics should not inﬂuence model decisions.
Key challenge: generating valid, intuitive and diverse fairness constraints A key challenge for ensuring individual fairness is deﬁning the similarity notion φ , which can often be contentious, since fairness is a subjective concept, as well as highly task dependent [6, 7].
In text classiﬁcation, most existing works have cast similarity in terms of word replacement [5, 8–10].
Given a sentence s , a similar sentence s (cid:48) is generated by replacing each word in s , that belongs to a list of words
A i indicative of a demographic group i , by a word from list
A i (cid:48) , indicative of another group i (cid:48) (cid:54) = i .
This approach has several limitations: (i) it relies on exhaustively curated word lists A i of sensitive terms, (ii) the expressivity of the generated pairs is limited to word replacements, and (iii) many terms are only indicative of demographic groups in speciﬁc contexts, hence directly replacing them with other terms will not always result in a similar pair ( s, s (cid:48) ) according to human intuition. Indeed,
[Figure]
word rules can often produce sentence pairs that differ in an axis not relevant to g., by “white house” with “black house”).
In addition, they can generate asymmetric counterfactuals [5] ( s, s (cid:48) ) that look similar but do not warrant similar treatment.
For example, in the context of toxicity classiﬁcation, the text “The movie is so old” may not be considered toxic while “The movie is so gay” clearly is.
This work: generating fairness speciﬁcations for text classiﬁcation
The central challenge we consider in this work is generating a diverse set of input pairs that aligns with human intuition about which inputs should be treated similarly in the context of a ﬁxed text classiﬁcation task.
We address this challenge via a three-stage pipeline (Fig. 1).
First, we start from a dataset D and generate a set
C w of candidate pairs ( s, s (cid:48) )
by applying word replacement to sentences s ∈ D .
Second, to improve the diversity of pairs, we extend C w to a larger set C e by borrowing ideas from unsupervised style transfer.
We change markers of demographic groups, e.g., “women” or “black people” in sentences s ∈ D by replacing the style classiﬁer in modern unsupervised style transfer methods [11, 12] with a classiﬁer trained to identify mentions of demographic groups.
In addition, we add pairs from
GPT-3 [13], prompted to change markers of demographic groups for sentences in D in a zero-shot fashion.
Finally, to identify which of the generated pairs align with human intuition about fairness, we design a crowdsourcing study in which workers are presented with candidate pairs and indicate if the pairs should be treated similarly for the considered classiﬁcation task or not.
We employ active learning similar to [14] to train a BERT-based [15] classiﬁer ˆ ϕ to recognize pairs that should be treated similarly using a limited amount of human feedback and obtain a ﬁltered set of pairs ˆ C r ⊆
C e .
Our pipeline can be used in the context of most text classiﬁcation tasks and in this work we instantiate it in the context of toxicity classiﬁcation using a large dataset for online content moderation.
Main contributions We make the following contributions: (i) we introduce a method for generating datasets of diverse candidate pairs for individual fairness speciﬁcations, leveraging GPT-3 and unsupervised style transfer to modify demographic attributes mentioned in sentences; (ii) we show that human feedback can be used to train a classiﬁer which automatically identiﬁes pairs that align with human fairness intuitions for a considered downstream task; (iii) we instantiate our framework in the context of toxicity classiﬁcation, demonstrating that the proposed pairs are more diverse than word replacement pairs only and that crowdsourcing workers agree with more than 75% of them.
Related Work
Bias in NLP Early work on bias in NLP has focused on unwanted correlations between the word embeddings of identiﬁers for protected demographic groups and unrelated categories such as occupations [16, 17].
Recently, language models have been found to harbor stereotypical biases [10, 18–20].
Speciﬁc to text classiﬁcation, identity terms such as “gay” and explicit indicators of gender have been shown to impact the outputs of classiﬁers trained to identify toxic comments [8] or to predict a person’s occupation from their biography [4].
Olteanu et al.
[21] demonstrate that human perceptions of the quality of a toxicity classiﬁer can depend on the precise nature of errors made by the classiﬁer, as well as the annotators’ previous experiences with hate speech.
Blodgett et al.
[22] recommend authors to explictly consider why, how and to whom the biases they identify are harmful.
Language models for data augmentation Ross et al.
[23] automatically create contrast sets [24] with a language model perturbing sentences based on control codes, while Rios [25] use style transfer to change the dialect of African-American Vernacular English tweets to Standard American English to evaluate the sensitivity to dialect of toxicity classiﬁer.
Hartvigsen et al.
[26] use language models to generate a balanced dataset of benign and toxic comments about minority groups to combat classiﬁers’ reliance on spurious correlations between identity terms and toxicity.
Meanwhile, Qian et al.
[27] train a perturber model to imitate human rewrites of comments that modify mentions of demographic groups, and demonstrate that their perturber can be used to reduce demographic biases in language models.
However, this approach is limited by its reliance on expensive human rewrites and is only used for perturbations along ﬁxed demographic axes such as gender.
Learning fairness notions from data Ilvento [28] provides an algorithm to approximate individual fairness metrics for N datapoints in O ( N log N ) queries, which can be practically infeasible.
Meanwhile, Mukherjee et al.
[29] suggest training a classiﬁer to predict binary fairness judgments on pairs ( s, s (cid:48) ) in order to learn a fairness metric φ , but restrict themselves to Mahalanobis distances on top of a feature representation ξ ( s ) , limiting their expressive power.
In contrast to our work, these works do not validate their learned fairness notions with human feedback.
To that end, Cheng et al.
[30] present an interface to holistically elicit stakeholders’ fairness judgments, whereas Wang et al.
[31] aim to learn a bilinear fairness metric for tabular data based on clustering human annotations.
Method
This section presents our end-to-end framework for generating and ﬁltering valid candidate pairs for individual fairness speciﬁcations.
In Sec.
3.1 we expand on existing word replacement deﬁnitions of individual fairness in text classiﬁcation [5] by implementing three different ways to modify markers of demographic groups mentioned in a sentence s .
Then, in Sec.
3.2 we leverage human feedback to learn an approximate similarity function ˆ ϕ to identify a set of relevant constraints ˆ C r ⊆ C e .
3.1 Expanding fairness constraints
Word Replacement First, we enrich the word replacement method by using the extensive lists of words associated with different protected demographic groups presented in Smith et al.
The pool of terms is substantially larger than the 50 identity terms from Garg et al.
We modify markers of group j in a comment s by replacing all words on the respective list of words associated with group j with words from the list associated with the target group j (cid:48) .
Unsupervised Style Transfer Second, we use an unsupervised style transfer approach based on prototype editing (see [32] for an extensive review) to transform markers of a demographic group j in a sentence s to markers of another demographic group j (cid:48) , creating a new sentence s (cid:48) .
Prototype editing identiﬁes markers a of a source style A in a text s , and substitutes them by markers a (cid:48) of a target style A (cid:48) .
Our approach leverages that modern prototype editing algorithms utilize saliency methods in combination with a style classiﬁer to identify markers of style, and instead uses a RoBERTa-based [33] classiﬁer c trained to identify sentences that mention speciﬁc demographic groups j .
Combining ideas from [11] and [12], we transform a sentence s to mention demographic attribute j (cid:48) instead of j by iteratively masking tokens with large impact on the likelihood p c ( j | s m ) (initially starting with s m = s ) until we reach a certain threshold, and ﬁlling the masked tokens using a BART-based [34] group-conditioned generator g ( s m , j (cid:48) ) trained to ﬁll masks in sentences about group j (cid:48) .
The unsupervised style transfer approach is likely to reproduce terms encountered during training, helping it to pick up on rare demographic terms that are particular to its training distribution which can be chosen to equal the training distribution for downstream tasks.
In addition, unlike concurrent work by Qian et al.
[27], unsupervised style transfer only requires labels y j ( s ) indicating the mention of demographic group j in a sentence s rather than expensive human-written examples of demographic group transfer.
This allows us to modify mentions of demographic groups across axes like gender, religion and race, rather than restricting ourselves to changes within these axes.
GPT-
Lastly, we leverage GPT-3 [13] to transform markers of protected demographic groups.
We consider three methods: using GPT-3 standard mode and GPT-3 edit mode to rewrite sentences mentioning group j to mention group j (cid:48) in a zero-shot fashion, as well as postprocessing sentences generated by word replacement to ﬁx logical and grammatical inconsistencies with GPT-3 edit mode.
To ensure that mentions of demographic group j were indeed replaced by j (cid:48) going from s to s (cid:48) , we use the same group-presence classiﬁer c as for the unsupervised style transfer approach to heuristically identify successful group transfer and discard pairs ( s, s (cid:48) ) for which group transfer failed, for all three of our approaches.
Implementation details are described in App.
C and App.
E contains examples.
3.2 Learning the similarity function
In order to evaluate to what extend the proposed similarity criteria align with human intuition, we conduct a crowdsourcing study, described in more detail in Sec. 4, to obtain labels ϕ
( s, s (cid:48) ) which indicate whether a pair ( s, s (cid:48) ) should be treated similarly for the sake of individual fairness ( ϕ ( s, s (cid:48) )
= 0 ) or not ( ϕ ( s, s (cid:48) )
= 1 ).
We train a BERT-based [15] probabilistic model p ˆ ϕ ( s, s (cid:48) ) that predicts values of the similarity function ϕ ( s, s (cid:48) ) for pairs ( s, s (cid:48) )
∈ C e , and approximate the similarity function φ as ˆ ϕ
( s, s (cid:48) ) :
= 1 ⇔ p ˆ ϕ ( s, s (cid:48) )
> t for a given classiﬁcation threshold t .
To make optimal use of costly human queries, we employ active learning when training the classiﬁer ˆ ϕ , selecting pairs to label based on the variation ratios 1 − max y p ( y | x ) with p estimated similar to Grießhaber et al.
[14], based on Dropout-based Monte-Carlo [35, 36] applied to our model’s classiﬁcation head.
Concretely, we iteratively select new unlabeled training data D i ⊂ C e \ (cid:83) j<i D j with | D i | = 1000 , based on the variation ratios, query labels for D i , and retrain ˆ ϕ on D i .
As different annotators can disagree about whether two sentences s and s (cid:48) should be treated similarly, we use a majority vote for evaluation.
Inspired by Chen et al.
[37]’s approach for dealing with noise in crowdsourcing, we use a single human query per pair ( s, s (cid:48) ) during active learning, and relabel pairs that are likely to be mislabeled after active learning has concluded.
See App.
D for more details.
When learning ˆ ϕ is completed, we can deﬁne the set of ﬁltered constraints ˆ C r = { ( s, s (cid:48) )
∈
C e : ˆ ϕ ( s, s (cid:48) )
= 0 } .
Experiments
In this section, we experimentally evaluate our framework.
Our key ﬁndings are: (i) the pairs generated by our method are more diverse compared to word replacement pairs only (Sec. 4.2), while mostly aligning with human intuition about fairness (Sec. 4.3) and (ii) the underlying similarity function ϕ can be approximated by active learning from human judgements (Sec. 4.4).
4.1 Dataset and setup
We focus on toxicity classiﬁcation on the Jigsaw Civil Comments dataset [38].
The dataset contains around 2 million online comments s with labels y ( s ) indicating toxicity.
We focus on a subset
D (cid:48)
⊂ D with labels A j ( s ) that indicate the presence of group j in s for training our group-presence classiﬁer c , and only consider comments s that consist of at most 64 tokens.
We construct a set C e of 100 , 000 constraints applying our different generation approaches to D (cid:48) 1 .
More details on the generation and exact composition of C e , as well as example pairs ( s, s (cid:48) ) , can be found in App.
C.
Throughout this section, whenever we report fairness for a classiﬁer f , we refer to the proportion of pairs ( s, s (cid:48) ) in a test pool of similar pairs for which f ( s ) =
f ( s (cid:48) ) rather than f ( s ) (cid:54) =
f ( s (cid:48) ) .
4.2 Diversity of generated fairness constraints
To validate that our candidate constraint set C e is more diverse than word replacement on its own, we train 4 different toxicity classiﬁers, using Counterfactual Logit Pairing (CLP) [5] to empirically enforce different constraint sets C e , C 1 , C 2 , C 3 .
Here C e corresponds to the full constraint set, as described in Sec. 3.1.
while the other constraint sets have the same size as C e , but contain pairs generated by one method only.
In particular, the pairs in C 1 were generated by word replacement using the 50 identity terms from Garg et al.
[5] 2 , the pairs in C 2 were generated by word replacement, using the larger list of terms of Smith et al.
[20], and the pairs in C 3 were derived by style transfer. [Table] [Table]
We then cross-evaluate the performance of the 4 classiﬁers trained with these constraint sets in terms of their test-time fairness according to each of the 4 fairness criteria, and their balanced accuracy.
The results in Table 1 show that each classiﬁer achieves high fairness when evaluated on the set of pairs corresponding to the constraints used during its training (numbers in italics) while performing worse on other constraint pairs.
While this indicates that adherence to fairness constraints does not always generalize well across our generation methods, we note that training on style transfer pairs ( C e or C 3 ) generalizes substantially better to C 2 than training on different word replacement pairs ( C 1 ; see the numbers in bold).
More details can be found in App.
C.
4.3 Relevance of generated fairness constraints
To validate that the generated fairness contraints are relevant and intuitive, we conducted a human evaluation with workers recruited via Amazon MTurk.
The workers were presented with pairs ( s, s (cid:48) ) consisting of a comment s from the Civil Comments dataset, as well as a modiﬁed version s (cid:48) and asked about whether they believe that the two comments should be treated similarly and whether they believed that the average American shared their opinion.
Treatment was framed in terms of toxicity classiﬁcation for the sake of content moderation, ensuring that we verify the relevance of the learned notions relevant to this speciﬁc task.
The workers were also asked whether the demographic group was transferred correctly from a given j to a given j (cid:48) , whether the content of s has been preserved in s (cid:48) apart from the demographic group transfer, and whether there are differences in factuality and grammaticality between s and s (cid:48) .
We collected human feedback for a set S containing a total of 720 pairs ( s, s (cid:48) ) with 240 each being produced by our style transfer approach, GPT-3 in a zero-shot fashion, and word replacement using the list from [5] as for C 1 .
These 240 pairs per method were split into 80 pairs for each of the axes male ↔ female, christian ↔ muslim and black ↔ white.
Each pair ( s, s (cid:48) ) was shown to nine different workers.
Further details can be found in App.
B.
Table 2 shows that all three methods mostly produce relevant fairness constraints, according to a majority of annotators.
At the same time, they generally successfully modify the mentioned demographic group, and preserve content, factuality and grammaticality.
While word replacement generally performs better in terms of group transfer and content preservation, it only has a small advantage in terms of relevance to fairness, perhaps due to its worse performance in terms of factuality:
[Table]
we found examples in which word replacement changed "white house" to "black house"; or Obama is referred to as "white" rather than "black".
These pairs were not seen as fairness constraints by most annotators and judged badly in terms of preserving factuality.
See B.1 for more detailed results.
4.4 Learning the similarity function
We employed our active learning approach to efﬁciently train a classiﬁer ˆ ϕ from relatively few human judgments [Table] with the goal of using it to identify pairs that represent actual fairness constraints on the remaining pool of candidates.
We conducted 6 steps of active learning with 1000 queries each and discarded failed queries, ending up with a total of 5490 labeled pairs (( s, s (cid:48) ) , ϕ ( s, s (cid:48) )) .
Details on our model architecture and other hyperparameters can be found in App.
D. We evaluate our learnt classiﬁer on a test set T consisting of 500 randomly selected pairs from C e for which ﬁve annotators were asked about the average American’s fairness judgment.
Because 78 . 8% of the pairs ( s, s (cid:48) ) in T represented fairness constraints ( ϕ ( s, s (cid:48) )
= 0)
according to the majority of annotators, we report Balanced Accuracy (BA), in addition to standard accuracy (ACC) and the true positive and negative rates (TPR and TNR).
Table 3 displays these metrics for classiﬁers resulting from our active learning method for different classiﬁcation thresholds t and with and without relabeling.
We observe that ˆ ϕ performs substantially better than random, achieving BA of 66 .
7% when used with an aggressive classiﬁer threshold t .
The table also validates our relabeling approach: after observing that our classiﬁer was biased towards predicting ϕ ( s, s (cid:48) )
= 0 , we collected two additional labels for 500 pairs ( s, s (cid:48) ) for which both the human and the predicted label were equal to zero ( ˆ ϕ ( s, s (cid:48) )
=
ϕ
( s, s (cid:48) )
= 0 ), selected based on the variation ratios.
47% of these pairs received a majority vote of ϕ ( s, s (cid:48) )
= 1 , showing that our approach correctly identiﬁed pairs that were likely to be mislabeled.
For the balanced classiﬁcation thresholds t = 0 . 5 , retraining our classiﬁer on the updated majority votes also substantially increased TPR at little costs to TNR.
According to a qualitative evaluation, most of the sentence pairs ( s, s (cid:48) ) predicted to not represent fairness constraints ( ˆ ϕ ( s, s (cid:48) )
= 1) had the words "boy" or "man" replaced by terms denoting identity membership.
Such sentence pairs were often not seen as fairness constraints by our annotators, as the inclusion of the identity term can be interpreted as aggressive or mocking.
ˆ ϕ also successfully identiﬁed sentence pairs ( s, s (cid:48) ) for which s (cid:48) was unrelated to s , that were sometimes produced by
GPT-3, as not representing fairness constraints.
Additional results and details can be found in App.
D.
Conclusion
We proposed a framework for producing expressive and intuitive speciﬁcations for individual fairness in text classiﬁcation.
We experimentally demonstrated that our constraints are indeed more expressive than previous constraints based on word replacement and validated that most of the generated fairness constraints were relevant in the context of toxicity classiﬁcation according to human annotators.
In addition, we used active learning to demonstrate that human fairness judgments can be predicted using limited amounts of training data.
In future work we plan to utilize the generated ﬁltered constraints to train a fair downstream toxicity classiﬁer with better trade-off between accuracy and fairness.
Further work could explore approaches similar to ours for other NLP tasks with discrete outcomes beyond classiﬁcation, as well as for evaluating robustness to other forms of intuitively relevant perturbations.
Acknowledgements
We would like to thank Dan Hendrycks, Mislav Balunovi´c, Afra Amini and Dominik Stammbach for helpful comments and discussions during early stages of this work.
We would also like to thank the anonymous reviewers for their comments and feedback.
Florian Dorner is grateful for ﬁnancial support from the Max Planck ETH Center for Learning Systems (CLS) received during part of this work.
Nikola Konstantinov’s contributions to this publication were made possible by an ETH AI Center postdoctoral fellowship
A Ethical Considerations
Our human evaluation experiments involving workers from Mechanical Turk were reviewed and approved by the ETH Zurich Ethics Commission as proposal EK 2022-N-117.
Workers on Mechanical
Turk were warned that they might be shown offensive comments as part of our study and were able to opt out of participating in our study at any time.
We also made sure that the per-task compensation was sufﬁciently high to result in a hourly compensation exceeding the US federal minimum wage.
More details on our human evaluation experiments can be found in App.
B.
While we believe that our results show that learning more precise fairness notions by involving human feedback is a very promising area of research, we caution against directly using our learnt similarity classiﬁer φ for evaluating fairness in high-stakes real-world applications of toxicity classiﬁcation.
First, our results show that there is substantial disagreement between different survey participants about which pairs ( s, s (cid:48) ) require equal treatment by a fair classiﬁer.
While resolving these disagreements via a majority vote is a natural choice, other approaches may be desired in some contexts (for e.g., enforcing equal treatment whenever at least one participant believes it is required).
Second, our survey participants may have geographic biases and are neither direct stakeholders, nor experts in discrimination law and hate speech.
Given that our learning approach shows promising signs of being able to improve upon existing fairness deﬁnitions despite large amounts of disagreement, which is likely to be less common for actual stakeholders and experts, we recommend using it in conjunction with fairness judgments provided by application-speciﬁc experts and stakeholders.
B Further Details on Human evaluation
In order to participate, workers had to live in the US and be above 18 years old in addition to being experienced with MechanicalTurk (having completed more than 5000 HITs 3 and having a good reputation ( 97% acceptance rate across all of the worker’s HITs).
Workers were warned about the potentially offensive content of some of the comments show in the study by the following statement: "Please note that this study contains offensive content.
If you do not wish to see such content, please withdraw from the study by leaving this website."
and were also told that they could withdraw from the study at any later point: "You may withdraw your participation at any time without specifying reasons and without any disadvantages (however, you will not get paid for the current HIT in case you withdraw before completing it)".
After encountering a high prevalence of bots, malicious workers or workers that fundamentally misunderstood our task instructions during pilot experiments, we had workers pass a qualiﬁcation test by providing correct answers for nine out of ten queries ϕ
( s, s (cid:48) ) for pairs that were handdesigned to have a relatively obvious correct answer.
We validated these hand-designed pairs in a separate experiment, querying workers about ϕ ( s, s (cid:48) ) for 11 pairs ( s, s (cid:48) ), and asking them to verbally explain each of their decisions, paying a total of $1 . 83 .
We only included hand-designed pairs in the qualiﬁcation test if at least eight out of ten workers produced the intended answer during this experiment, and no worker brought forward convincing arguments against this answer being correct.
Workers were paid $0 .
83 for a battery of 11 queries ϕ
( s, s (cid:48) ) for the Active learning and the test set
T and ﬁrst $1 . 20 , later $1 . 30 for 11 queries about all questions presented in Table 2.
In both cases, workers had to explain one of their answers verbally, and one of the queries was a attention check pair ( s, s (cid:48) ) with a relatively obvious correct answer constructed in the same way as for the qualiﬁcation tests.
Blocks of queries with wrong answers to the attention check question or (partially) incoherent verbal explanations were manually reviewed, and thrown out in case we were not able to ﬁnd evidence that the worker had correctly understood the task in which case the worker was not paid.
Workers were made aware of this by the following statement in the task description: "To ensure workers pay attention, we include pairs of comments for which there is a clear correct answer at random positions.
Please make sure to carefully read every comment and answer diligently, as your HIT will be rejected if your answer for one of these pairs of comments is wrong, in which case you will not receive a payment for the HIT.
We will also ask you to produce an explanation for one of your answers.
Please write at least one sentence and make sure your explanation is coherent with the answer you gave, as your HIT might be rejected otherwise in which case you will not receive a payment for the HIT.".
According to https://turkerview.com, a tool used by many professional workers on MechanicalTurk, we paid workers an average hourly rate of $16 .
17 , clearly exceeding the US minimum wage.
While this is likely an overestimate, as not all workers use turkerview, the hourly rate is so high, that we still exceed the minimum wage for workers taking twice as long as the average worker using turkerview.
Workers were introduced to the task as follows: "We are investigating fairness in toxic comment classiﬁcation and are looking for pairs of comments with similar content that should be treated the same by a content moderator (if one of them was removed, the other one should be removed too).
You will be shown pairs of comments and be asked whether the comments are similar in various ways and whether you would consider it unfair for a content moderator to remove one of the comments in each pair but not the other.".
The queries read as follows for the ﬁrst task:
• Without further context, would the average american consider it unfair or acceptable for a content moderator to delete one of the shown comments but not the other?
– It would be unfair for one of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
– If comment a) was deleted, comment b) should be deleted as well, but not the other way around.
(Deleting comment b) would be more justiﬁable) – If comment b) was deleted, comment a) should be deleted as well, but not the other way around.
(Deleting comment a) would be more justiﬁable) – It would be acceptable to delete either of the comments without deleting the other.
(Treating the comments differently either way would be ﬁne)
and as follows for the second one (Placeholders for group terms are presented in Italics .
The answer summarized in Table 2 is highlighted in bold ):
• Without further context, would you consider it unfair or acceptable for a content moderator to delete one of the shown comments but not the other?
– It would be unfair for one of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
– If comment a) was deleted, comment b) should be deleted as well, but not the other way around.
(Deleting comment b) would be more justiﬁable) – If comment b) was deleted, comment a) should be deleted as well, but not the other way around.
(Deleting comment a) would be more justiﬁable) – It would be acceptable to delete either of the comments without deleting the other.
(Treating the comments differently either way would be ﬁne) •
What do you expect the average American to answer to the previous question?
– It would be unfair for one of the two comments but not the other to be deleted (Both comments ought to be treated the same way, no matter what)
– If comment a) was deleted, comment b) should be deleted as well, but not the other way around.
(Deleting comment b) would be more justiﬁable) – If comment b) was deleted, comment a) should be deleted as well, but not the other way around.
(Deleting comment a) would be more justiﬁable) – It would be acceptable to delete either of the comments without deleting the other.
(Treating the comments differently either way would be ﬁne) •
Do you consider one of the two comments to be more factually correct than the other?
– Comment a – Comment b – Neither comment / too little context to tell • Do you consider one of the two comments to be more grammatically correct than the other?
– Comment a – Comment b – Neither comment • Is comment a) about group a and comment b) about group b ?
– Yes
– No, comment a) is not about group a – No, comment b) is not about group b – No, neither • Apart from differences related to group a and group b , are both comments similar in terms of content?
– Yes, they are almost the same.
– They are somewhat similar, but differ in some additional details.
– There is an important additional difference between the comments’ content
B.1 shows the results of the human evaluation on our test set S split along the axis of attribute transfer, rather than generation method as in 2.
Along with the results in Table 2 they show that despite the general agreement about the relevance of the generated fairness constraints, there is substantial disagreement between annotators when it comes to deviations from the most common answer across all comments.
In all cases, the fraction of comments with majority vote equal to that answer is substantially higher than the overall fraction of these votes across all comments and annotators.
The same is true for our set T of 500 randomly selected pairs from C e for which we only asked about the average American’s fairness judgment: 70 .
9% of the annotations were ϕ
( s, s (cid:48) )
= 0 , while the same was true for 78 .
8% of the per-comment pair majority votes.
[Table]
C Further details on style transfer
Unsupervised style transfer To transform markers of demographic groups in sentences, we ﬁrst ﬁnetune a Multi-headed RoBERTa-based [33] classiﬁer c to predict labels
y j indicating the presence of markers of a demographic group j from a list of protected demographic groups J in a sentence s .
We use the population labels ("Black", "Male", "Heterosexual", "Muslim",etc.)
that are provided for a subset of the Civil comments dataset.
The group-presence classiﬁer c is based on the roberta-base model, followed by a linear layer with 768 neurons applied to the output embedding of the ﬁrst token only, a Tanh layer, another linear layer mapping to a single dimension, and a Sigmoid layer.
We train c for 3 epochs with a batch size of 16 and use the Adam optimizer [39] with learning rate 0 .
to optimize the binary Cross Entropy loss, reweighed by relative label frequency in the dataset.
Table C.1 shows the balanced accuracy on the test set for all demographic groups in the dataset.
For our downstream applications of c , we restrict ourselves to the demographic groups for which the classiﬁer c ’s balanced accuracy is above 90% .
Furthermore, we also exclude the group labeled "mental illness" because the word replacement lists we used lack a clear analogon.
Then, we ﬁnetune a BART-based [34] generator g on a mask-ﬁlling task on the same data: For every data point s , we sample a group from the set of demographic groups j mentioned in s , i.e.
{ j : y j ( s ) = 1 } , skipping sentences s for which no group j meets this criterion.
Inspired by [11] we mask all of s ’s tokens that have an above-average attention value for the 11th layer of the classiﬁer c , merge consecutive mask tokens into one, and prepend the name of the sampled group j to the masked sentence before fedding it to the generator g .
The generator g is then ﬁnetuned to reconstruct s using token-wise Cross Entropy.
[Table]
The BART-based generator g is trained starting from the pretrained facebook/bart-large model for a single epoch with batch size 4, again using Adam and a learning rate of 0 . 00001 .
For ﬁlling in masked sentences, we pick the completion with the largest difference in the classiﬁer c ’s pre-sigmoid activation for the target and source demographic groups j (cid:48) and j among candidate sentences produced by a beam search generation using the generator g with width 5 .
To transfer an example s from mentioning group j to mentioning group j (cid:48) , we follow [12] and iteratively mask the token for which masking reduces p c ( y j | x ) the most, until we reach a threshold of p c ( y j | x ) < 0 .
We use this approach rather than the attention-based masking from [11] because of the lack of theoretical motivation for using attention to identify important features [40], and because attention scores are the same for all of our model’s group-presence prediction heads, rather than speciﬁc to a particular group j .
Then, we prepend a verbal representation of label j (cid:48) to s to form a prompt p , and generate a sentence s (cid:48) as g ( p ) .
Word replacement
Our word replacement approach is based on the list of words provided in [20]:
Given a sentence s mentioning demographic group j and a target attribute j (cid:48) , we replace all words in s that are on the list associated with j with random words from the list associated with j (cid:48) , replacing nouns with nouns and descriptors with descriptors whenever possible, and nouns with descriptors otherwise.
The full list of words we used for word replacement is displayed in Table E.1.
GPT-
We accessed GPT-3 using OpenAI’s API 5 .
For our ﬁrst approach, we used the "text-davinci001" version of GPT3 in a zero-shot manner with the prompt: "Please rewrite the following sentence to be about j (cid:48) rather than j :" followed by a new line and the targeted sentence s .
The second approach was based on the beta-version of GPT-3’s editing mode 6 .
Here, s (cid:48) is produced using the model "text-davinci-edit-001" with the instruction "Rewrite the text to be about j (cid:48) rather than j ".
Lastly, we used to same model in conjunction with word replacement:
First, we generated a candidate sentence s (cid:48)(cid:48) using the procedure described in the word replacement section.
Then, in order to ﬁx issues caused by the context-blindness of the word replacement approach, we postprocessed s (cid:48)(cid:48) using "text-davinciedit-001" with the instruction "Fix grammatical errors and logical inconsistencies" to produce s (cid:48) .
We used temperature = 0 .
7 and top_p = 1 in all our approaches and used max_tokens = 64 for "text-davinci-001" to control the length of the modiﬁed sentence s (cid:48) .
Post-ﬁltering For all three approaches, we performed a post-ﬁltering step to reduce the prevalence of unsuccesful attempts at demographic group transfer in our set of constraints C e .
Given a pair ( s, s (cid:48) ) of an original sentence and a modiﬁed version, we only include it in our set of constraints C e , if the classiﬁer probability p c ( y j (cid:48) | s (cid:48) ) for label j (cid:48) is below 0 .
5 and the classiﬁer probability p c ( y j | s (cid:48) ) for label
j is above 0 .
As mentioned in Sec.
4.1, we attempt to produce modiﬁed comments s (cid:48)
j (cid:48) mentioning group j (cid:48) for each s in D (cid:48) for all demographic groups j with y j ( s ) = 1 and all possible target groups j (cid:48) for word replacement and style transfer.
For GPT-3, we attempted a total of 75 generations for each of our three
[Table]
generation modes per axis pair of demographic groups ( j, j (cid:48) ) and direction of group transfer, with the source sentences s randomly selected among the sentences with label j in D (cid:48) .
For constructing the secondary test set S , we attempted more generations for the axes male ↔ female, christian ↔ muslim and black ↔ white, homosexual ↔ heterosexual.
The latter axis was left out of S because we found that the rate of successful generations was too limited.
We generated a maximum of 2250 attempts up until a total of 250 successful generations (post-ﬁltering step passed) for GPT-3’s zero-shot mode, a maximum of 750 until to a total of 100 successful generations for GPT-3’s edit mode, and up until a total of 100 successful generations for GPT-3 based postprocessing of word replacement.
Table C. shows the overall amount of generated pairs per method.
As an additional experiment to validate the increased diversity of our constraint set C e we train a similarity classiﬁer 7 ˆ ϕ , on C e to distinguish pairs ( s, s (cid:48) ) generated by word replacement from pairs generated by style transfer or GPT-3.
Training on 100000 examples without label noise, we are able to achieve 91 .
6% test accuracy on a balanced test set, suggesting that there is a meaningful difference between pairs generated by word replacement and the rest of the constraint candidates C e .
D Further details on learning similarity functions
First, Proposition D.1 below establishes that robustness with respect to a binary similarity function ϕ , i.e. ϕ ( s, s (cid:48) )
=
⇒ f
( s ) =
f ( s (cid:48) ) , can fully capture the deﬁnition of individual fairness as
Lipschitz-Continuity proposed by Dwork et al.
[6] for deterministic classiﬁers f .
Proposition D.1.
Given a metric d : X × X → R , a binary metric d b : Y × Y → { 0 , 1 } and a constant L > 0 , there exists a similarity function ϕ :
X × X → { 0 , 1 } such that a function f : ( X, d ) → ( Y, d b ) is Lipschitz-Continuous with constant L if and only if ϕ ( x, x (cid:48) ) ≥ d b ( f ( x ) , f
( x (cid:48) )) for all x, x (cid:48) ∈ X .
Proof.
Deﬁne ϕ
( x, x (cid:48) ) :
= 1 { Ld ( x, x (cid:48) ) ≥ 1 } .
Then whenever d b ( f ( x ) , f ( x (cid:48) ))
= 1 , we have d b ( f ( x ) , f ( x (cid:48) ))
= 1 ≤ ϕ ( x, x (cid:48) )
if and only if d b ( f ( x ) , f ( x (cid:48) )) ≤ Ld ( x, x (cid:48) ) .
But if d b ( f ( x ) , f ( x (cid:48) ))
= 0 , the Lipschitz inequality is allways true.
Now, assume that f is not Lipschitz: Then, there exist x, x (cid:48)
∈
X such that 1 = d b ( f ( x ) , f ( x (cid:48) )) >
Ld ( x, x (cid:48) ) , implying
= ϕ
( x, x (cid:48) ) <
d b ( f ( x ) , f ( x (cid:48) ))
=
We use a BERT-based classiﬁer that acts on a pair ( s, s (cid:48) ) by ﬁrst tokenizing both s and s (cid:48) and padding the token representation to a length of 64 , concatenating these tokens and feeding the concatenated token representation into a pretrained bert-uncased-base model.
We then apply a linear layer with dropout ( p = 0 . 1 ) followed by a Tanh layer and a second linear layer with dropout ( p = 0 . 1 ) to obtain single dimensional logits, to which a sigmoid layer is applied before computing the binary
Cross Entropy loss and optimizing it using the Adam optimizer with learning rate 0.00001 applied to all model parameters.
We use BERT rather than more modern models such as RoBERTa [33] and Deberta [41], as we have found it to clearly outperform them for our task, plausibly because
BERT uses a next-sentence-prediction task during pretraining, which is structurally similar to our task of comparing two sentences.
Table D.1 demonstrates the advantage of using BERT, as well as concatenating token representations rather than learning based on the difference between separately produced BERT features for both s and s (cid:48) .
Unless stated otherwise, our Active Learning approach trains for ﬁve epochs on each queried block D i before selecting new data D i +1 to label.
[Table]
D.1 Synthetic Data
For active learning
[Table]
we freeze the underlying BERT model during the active learning selection and only apply MC-Dropout on the level of the classiﬁer head, similar to [14]
[Table]
but unlike them we do not use BALD [42] and instead approximate p ( y | s, s (cid:48) ) averaging the models’ predicted probabilities p ˆ ϕ
( y | s, s (cid:48) , w ) for 50 sampled dropout masks w .
We call this approach LC-UNC and experimented with various alternative selection criteria.
Unlike LC-UNC, LC directly approximates 1 − max y p ( y | s, s (cid:48) ) using a single forward pass through the ˆ ϕ with deactivated dropout.
BALD is the approach from [14], while VARRA and Majority approximate 1 − max y p ( y | s, s (cid:48) ) using MC-Dropout differently than LC-UNC:
In Majority, p ( y | s, s (cid:48) ) is approximated as the fraction of dropout samples w for which ˆ ϕ = 1 , while VARRA averages 1 − max y
p ˆ ϕ
( y | s, s (cid:48) , w ) over dropout samples w instead of averaging p ˆ ϕ ( y | s, s (cid:48) , w ) before applying the maximum operator.
In addition, the table contains the "automatic relabeling" condition in which D
i is selected from the whole of C e rather than just the previously unlabeled examples D i ⊂ C e \ (cid:83) j<i D j .
During training, pairs ( s, s (cid:48) ) that have been queried multiple times are labelled according to the majority vote of all queries, and as 0 .
5 in case of a tie.
We validate the efﬁcacy of our active learning approach for learning the similarity function ϕ
( s, s (cid:48) )
with a limited amount of noisy queries.
For this, we deﬁne two synthetic similarity functions ϕ
i : i ∈ { 1 , 2 } .
The ﬁrst, ϕ 1 is equal to zero, whenever a pair ( s, s (cid:48) ) was generated via word replacement and equal to one otherwise, as in the ﬁrst experiment from the previous section.
The second, ϕ 2 is equal to zero, whenever the group j of s that was removed and the added group j (cid:48) in s (cid:48) are within the same category of gender and sexuality, race, or religion, and equal to one otherwise.
For example, a pair ( s, s (cid:48) ) for which markers of "White people" in s were modiﬁed to markers of "Black people" in s (cid:48) would have ϕ 2 ( s, s (cid:48) )
= 0 , while ϕ 2 ( s, s (cid:48) ) would be one if the group was modiﬁed to "muslim" in s (cid:48) instead.
We simulate the label noise introduced by annotators’ disagreement by independently ﬂipping each label with probability p = 0 .
3 during training the similarity classiﬁer ˆ ϕ .
For training with 3 instead of one query per data point, we reduce the overall amount of training data from 10000 samples in C e to 3333 samples and reduce the probability of ﬂipping labels to p = 0 . 216 , simulating a majority vote.
In turn, the active learning approach selects 333 instead of 1000 data points for labeling in each of its ten steps in that scenario.
Table D.2 shows that active learning
[Table]
noticeably outperforms randomly sampling data points for our task, that there is no clear direct beneﬁt from employing multiple queries per pair ( s, s (cid:48) )
∈
C e over obtaining labels for previously unseen pairs, and that the LC-UNC setup is usually performing as well as or better than alternative selection criteria in the one-query per data point setting.
D.2 Human Evaluation
Tables D.3 and D.4 show additional results on the active learning from human feedback.
As above, we tested our approach using different ﬁltering thresholds t on the two test sets T (Table D.3) and S (Table D.4).
In the Retrain condition, the classiﬁer ˆ ϕ was trained for a single epoch on all labeled datapoints (cid:83) i<n D i in order to combat potential issues with catastrophic forgetting.
In the Retrain + Reweigh condition, the same was done, but the Cross Entropy loss was reweighed to balance the empirical label frequencies in (cid:83) i<n D i .
In the From Scratch setting, we train a new classiﬁer on (cid:83) i<n D i for 5 epochs from scratch without ﬁrst training it separately on any D i .
Again, datapoints are reweighed according to their empirical frequency in (cid:83) i<n D i in the From Scratch + Reweigh setting.
E Word Lists And Example Generations
Tables E.2–E.4 show 5 randomly example pairs ( s, s (cid:48) ) produced by our style transfer approach and
GPT-3 in zero-shot and edit mode.
Warning:
Some of the example texts contain offensive language.
[Table] [Table]
Reweigh
[Table]
[Table] [Table]
7 ± 4 .
AL
[Table]
= [Table] [Table] [Table]
1 .21 . 0 ± 2 .
AL + Relabel
[Table]
6 ± 0 .92 .
2 ± 0 .24 .
6 ± 1 .
AL + Relabel + Retrain
[Table]
[Table] [Table] 8 ±
[Table]
AL
[Table]
Relabel + Retrain + Reweigh
[Table]
[Table] [Table] [Table]
From Scratch t=0.
[Table] [Table]
[Table]
From Scratch + Reweigh t=0.
[Table]
[Table] [Table]
AL t=0.
6 ± 2 .82 .
3 ± 2 .38 .
2 ± 6 .
AL + Relabel
[Table]
2 ± 1 .85 .
5 ± 1 .30 . [Table]
AL
[Table]
[Table]
[Table]
0 .100 . 0 ± 0 . AL
[Table]
3 . 9 ± 0 .0 .
0 ± 0 .100 .
0 ± 0 .
From Scratch t=
[Table]
6 ± 0 .0 .
0 ± 0 .99 .
6 ± 0 .
From Scratch + Reweigh t=0.
5 . 4 ± 3 .1 .
6 ± 3 .50 .
8 ± 1 .
[Table]
gelical
[Table]
Hindu Hindu
Buddhist Buddhist
Atheist atheist, irreligious
Black
African-American, African, Black,
BIPOC
White
European-American,
European,
Caucasian, White
Asian
Chinese-American,
AmericanBorn Chinese, Chinese, IndianAmerican,
Indian,
KoreanAmerican, Korean, VietnameseAmerican, Vietnamese, AAPI
Latino
Cuban-American,Cuban,
Dominican-American,
Dominican,
Salvadoran-American,
Salvadoran, Guatemalan-American,
Guatemalan, Mexican-American,
Mexican, Filipina-American, Filipina, Filipino-American, Filipino,
Hispanic, Latinx, Latine, Latino,
Latina, Latin American
[Table]
’Hahaha, really?
Thousands of years of war and it was mostly white men, huh?
Read some history about non-European cultures to pierce your idyllic bubble.’ ’
Hahaha, really?
Thousands of years of war and it was mostly white European Christians, huh?
Read some history about non-European cultures to pierce your idyllic bubble’ ’
I would really like to see your source for the
Japanese fear of the Russians as being any kind of factor whatsoever.
For most of the
Japanese people, the Russians were not even on the radar screen!!’
’I would really like to see your source for the
Buddhist fear of the west as being any kind of factor whatsoever.
For most of the western people, the west and religion were not even on the radar screen!!’ ’
Yes: at least implicitly it endorses the old canard that the United States is a "Christian nation."’ ’
Yes: at least implicitly it endorses the old canard that the United States is a "white racist nation."’ ’
Please be careful what you label fake news.
News of shootings (Greenville?), migrants in
Sweden raising crime rates, Hispanics being rapists and murderers, illegal voting, etc those are examples of fake news....feet on couch?
not so much....’ ’
Please be careful what you label fake news.
Hundreds of shootings (Green Mountain?), migrants in Sweden raising crime rates, homosexuals being raped and murderers, illegal voting, etc those are examples of fake news....feet on couch?
not so much.’ ’
And often they are not so silent.
Gay friends have told me about various such clergy who are well known – only no one speaks out about it.’ ’
And often they are not so silent.
Gay friends have told me about various such men who are well known – only no one speaks out about it.’
[Table]
’OK, so now I ´ m confused.
Is it or is it not OK to mock a reporter ´ s physical handicap?
Call a Latina beauty queen "Miss Housekeeping"?
Post a mock video showing yourself clocking a woman you don ´ t like with a hard-hit golf ball?’ ’
Is it or is it not OK to mock a reporter ´ s physical handicap?
Call a Muslim beauty queen "Miss Housekeeping"?
Post a mock video showing yourself clocking a woman you don ´
t like with a hard-hit golf ball?’
"You caught me.
I confess.
I’ve been lying this whole time.
I’m not actually an atheist, of course I believe in a god.
I see him in the mirror every day !"
" You caught me.
I confess.
I’ve been lying this whole time.
I’m not actually a homosexual, of course I believe in a god.
I see him in the mirror every day !"
"Denying the existence of racism in America,(or most anywhere), is just ignoring reality.
The formal dicrimination has largely been curbed(with exception of using race to design
Congressional districts), but those wonderful ’family values’ of the white South still carry the mythology of inherent white superiority" ".
Denying the existence of racism in America,(or most anywhere), is just ignoring reality.
The formal dicrimination has largely been curbed(with exception of using race to design
Congressional districts), but those wonderful ’family values’ of the Hindu South still carry the mythology of inherent Hindu" ’Cite to "the Republicans, Trump being a Republican, who are against gays" please.
New one to me.
(Other than Trump being registered Republican of course, lol!)’ ’
Christian people are against gays.’
"You’d better watch out if you don’t claim
Christianity unto yourself like they do.
It is true that people harbor resentment over people whom they view as different.
It’s the only reason I can think of why they hate gay people."
" You’d better watch out if you don’t claim
Christianity unto yourself like they do.
It is true that people harbor resentment over people whom they view as different.
It’s the only reason I can think of why they hate Asians."
Original Comment
Modiﬁed Comment ’
So do they cover their faces?
This is about being able to identify women for practical purposes.
Makes sense to moi.’ ’
So do they cover their faces?
This is about being able to identify White people for practical purposes.
Makes sense to moi. ’ ’
What about
LGBTQI+M(xn)
W
[Equation]
omxn
[Equation]
Y
[Equation]
oxth
[Equation]
C [Equation] hxldrxn)O(thxrs)’ ’
Christianity is the best religion.
Christian people are the best people.
Jesus Christ was a man who lived a perfect life and died for our sins.
We should all be following Jesus because he is the author and ﬁnisher of our faith. ’ ’
Bust Judaism, Christianity and Islam are all
Abrahamic religions tracing their origins to the patriarch Abraham.
They are all related.’ ’
Bust Hispanic people are all the same.
They are all related. ’
"AO96 doesn’t allow men into womens rest rooms or locker rooms.
You do realize that trans people have been using the bathroom with you forever right?
Have you ever been injured by any of these people in all the years you have been using public facilities?"
"AO96 doesn’t allow white people into black rest rooms or locker rooms.
You do realize that trans people have been using the bathroom with you forever right?
Have you ever been injured by any of these people in all the years you have been using public facilities? "
’so.. I . have seen dozens of women in niqab in Toronto this week alone Almost every day
I see groups of them.
The problem with the niqab policy is that is was for such a silly narrow issue.
The niaqb should have been banned from public buildings and shopping malls.’ ’
so..
I . have seen dozens of black people in the niqab in Toronto this week alone Almost every day I see groups of them.
The problem with the niqab policy is that is was for such a silly narrow issue.
The niaqb should have been banned from public buildings and shopping malls. ’
[Figure]