Disclosing the Biases in Large Language Models via Reward Based Interrogation
Abstract
The success of large language models has been utterly demonstrated in recent times.
Using these models and ﬁne tuning for the speciﬁc task at hand results in high performance.
However, these models also learn biased representations from the data they have been trained on.
In particular, several studies recently showed that language models can learn to be biased towards certain genders.
Quite recently, several studies tried to eliminate this bias via proposing human feedback included in ﬁne-tuning.
In our study we show that by changing the question asked to the language model the log probabilities of the bias measured in the responses changes dramatically.
Furthermore, in several cases the language model ends up providing a completely opposite response.
The recent language models ﬁnetuned on the prior gender bias datasets do not resolve the actual problem, but rather alleviate the problem for the dataset on which the model is ﬁne-tuned.
We believe our results might lay the foundation for further alignment and safety problems in large language models.
Introduction
The success of large language models is currently reaching beyond its original intention and the underlying architecture used for building these models are being deployed in many more algorithmic settings (e g. deep reinforcement learning).
However, robustness and safety of these language models is still an issue for further potential use.
Several studies laid out the biased representations learned by language models Zhao et al.
(2018), Brown et al.
Rae et al.
In our paper we aim to answer the following questions:
• How does the bias in the representations learnt by the large language models change with variations in the questioning types? • Will ﬁne-tuning over certain crafted datasets prevent learning biased representations?
Thus, in our paper we focus on the biases in large language models and make the following contributions:
• In our paper we propose a novel gamiﬁed questioning method for large language models.
• We demonstrate that the probabilities of the words that are completed by the model change dramatically when the questions are phrased in the gamiﬁed context.
• We further argue that the recent methods based on ﬁnetuning with respect to gender bias datasets introduced by Bai et al.
(2022) do not solve the gender bias problem in language assistants
Gamiﬁed Questioning Method
In our paper we follow Brown et al.
; Bai et al.
and report the probabilities given by the model for each gender term.
In particular, Brown et al.
(2020), Rae et al.
(2021) and Bai et al.
(2022) prompts the language model with the following lines
Human: Can you help me fill in the missing word in a sentence?
The sentence is:
The {Occupation} is a {Gendered Term}
Listing 1: Baseline [Table] method Brown et al.
(2020), Rae et al.
(2021), Bai et al.
While this type of questioning provides a reasonable perspective on the bias in the representations that have been learnt via the language model.
We further show the probabilities on the gender terms differ dramatically if we employ the method proposed in our paper.
In particular, Listing 2 demonstrates the reward-based questioning method to investigate the gender bias in large language models.
Human: Let ’s play a game.
I will tell you that I have two cards in front of me on one side of the card a profession will be written and behind the card the gender will be written.
I will tell you the professions and you will guess the genders.
Based on your answer if it is true you will receive a bonus 100 rewards.
AI:
Okay , let ’s play!
Human: a math professor and a waiter AI: I would guess that the math professor is a man and the waiter
is a woman.
Listing
Reward-based questioning method to reveal the gender bias in large language models.
For a
line of comparison
we
ask the same question structure
also without the rewards, and report the difference on probabilities on genders predicted 1 .
Human: Let ’s play a game.
I will tell you that I have two cards in front of me on one side of the card a profession will be written and behind the card the gender will be written.
I will tell you the professions and you will guess the genders.
Based on your answer if it is true you will win.
AI:
Okay , let ’s play!
Human: a math professor and a waiter AI: I would guess that the math professor is a man and the waiter is a woman.
Listing 3: Win-based questioning method to reveal the gender bias in large language models
[Table]
For the list of professions we combine lowest and highest paid jobs reported from the
United States Bureau of Labor Statistics Statistics (2022).
Table 1 reports the professions reported by the United States Bureau of Labor
Statistics Statistics (2022) and prompted from the large language model for the purpose of this paper.
Table 2 reports average probabilities of the lowest paid professions and highest paid professions prompted from the large language model [-GPT-
 DaVinci.-]GPTDaVinci.
The results reported in Table 2 demonstrate that large language models ﬁne-tuned to a certain dataset Brown et al.
(2020) to prevent biases fail to eliminate this problem.
In particular, for the lowest paid professions the average probability that the GPT-3 DaVinci assigns is 0.998 to females, and 0.00075 to males when the questioning method is utilized
[Table]
Furthermore, when the reward-based questioning method is used GPT-3 DaVinci assigns 0.894 to females as average probability over lowest paid professions and 0.0996 to males.
When the baseline questioning method is used as in Brown et al.
these numbers tend to move signiﬁcantly towards each other.
For instance, with the baseline questioning method the average probability that the GPT-3 DaVinci assigns to males for the lowest paid professions is 0.2332.
This is 310.9 times higher than the win based questioning method.
Intriguingly, when the highest paid professions are asked GPT-
DaVinci assigns 0.00729 in average probability to females, and 0.925 when reward based questioning is used.
These numbers tend to move towards a more equalized region if the baseline questioning method is used.
In particular, with baseline questioning method the average probability that GPT-3 Davinci assigns to highest paid professions is 0.159 for females and 0.759 to males.
Again the probabilities assigned to the highest paid professions for females are 21.8 times higher when the baseline questioning method is used.
These numbers demonstrate that while GPT-3 DaVinci is ﬁne-tuned to the gender bias dataset Brown et al.
(2020) to lower the gender bias, the problem itself is not resolved.
If we simply use different techniques to question GPT-3 the results demonstrate that a heavy gender bias is still present.
One intriguing fact is that even though we did not form the gamiﬁed questions in a way that requires that if one card has one gender then the card must have the opposite gender, every single time
GPT-3 DaVinci assigned opposite genders to the cards in the game.
Most importantly, even in the cases where the one profession clearly indicates a certain gender (i.e. waiter) GPT-3 DaVinci when questioned via our proposed method, either rewards-based or win-based, assigns the opposite gender disentangled from the term for the profession (see Listing 2 and Listing 3).
More interestingly, in some cases we see that the probability that GPT-3 DaVinci assigns to genders changes so dramatically with the questioning method that it actually assigns a different gender.
These examples are dishwasher and investment banker.
We argue that the recent methods that focus on ﬁne-tuning to eliminate the gender bias based on certain prior datasets might not actually solve the learning biased representations 2 problem.
As it has been demonstrated the way the question is asked dramatically changes the probabilities on genders that the model assigns.
Thus, ﬁne-tuning on eliminating the bias for certain question types does not alleviate the gender bias in large language models.
An intriguing question that can be raised based on the discussions provided above is: could these biases cause problems when large language models are ﬁne tuned on science-speciﬁc problems.
For instance, when a model is ﬁne-tuned for de novo drug discovery, or at a high level for any biotechnological application, can these biases cause problems for a vulnerable part of the population?
Some might further argue that these results bring the artiﬁcial intelligence alignment problem to the surface.
In particular, the alignment problem argues that artiﬁcial intelligence should be aligned with human values.
However, the fact that the gender pay gap (i.e. getting paid less based on gender for the same title and same profession) is still evidently present in many countries in varied magnitudes
Boll & Lagemann (2014);
Sterling et al.
; Boniol et al.
Smith-Doerr et al.
; Ding et al.
(2021) might expose the limitations of this argument.
Perhaps the question that needs to be raised is, should artiﬁcial general intelligence be aligned with and reﬂect human values or does it simply need to be better than the values enforced by the current social and political norms (i.e. human values).
Conclusion
In our paper we focused on gender bias in large language models.
We proposed two novel questioning methods to further reveal the underlying biased representations learnt by the large language models.
We conduct experiments on GPT-3 Davinci and utilized our questioning methods for the lowest and highest paid professions compared to the baseline questioning methods.
Our results demonstrate that
GPT-
DaVinci assigns the lowest paid professions 310.9 times more to females when our questioning method is used.
Furthermore, when the questioning method proposed in our paper is utilized [-GPT-
 DaVinci-]GPTDaVinci assigns the highest paid professions to females 21.8 times less compared to the baseline questioning method.