Responsible Reasoning with Large Language Models and the Impact of Proper Nouns
Abstract
Language models with billions of parameters have shown remarkable emergent properties, including the ability to reason on unstructured data.
We show that open-science multi-lingual large language models can perform the task of spatial reasoning on two or more entities with signiﬁcant accuracy.
A socially responsible large language model would perform this spatial reasoning task with the same accuracy regardless of the choice of the names of the entities over which the spatial relationships are deﬁned.
However, we show that the accuracies of contemporary large language models are signiﬁcantly impacted by the choice of proper nouns even when the underlying task ought to be independent of the choice of proper nouns.
In this context, we observe that the conditional log probabilities or beam scores of language model predictions are not well-calibrated, and the scores do not discriminate between correct and wrong responses.
Introduction
Over the last four decades, cognitive psychologists and neurolingusitic studies have investigated the impact of proper names on human cognition with interesting outcomes (Bredart et al., 2002).
It is now widely accepted that human beings ﬁnd it more difﬁcult to recall personal names (Young et al., 1985, 1988) than other kinds of words, including relatively rare common nouns.
Several different explanations have been investigated for such a discrepancy, including the semantic tag nature of personal names without being descriptive (Fogler & James, 2007), the need to obtain a single correct label with no synonyms (Hanley, 2011), the larger set of possible phonologies (James & Fogler, 2007), and the frequency of word use in daily discourse (Kittredge et al., 2008).
Inspired by these results in human cognition, we seek to investigate the impact of personal names from different parts of the world on the linear spatial reasoning ability of large language models - a task that ought to be neutral to the choice of the names themselves.
The rise of the large language models promises to revolutionize differentiable approaches to natural language processing.
Models such as BERT (Devlin et al., 2018), T5 (Raffel et al., 2020),
GPT (Brown et al., 2020), OPT (Zhang et al., 2022), PALM (Chowdhery et al., 2022) and
BLOOM (BigScience, 2022) yield results close to the state-of-the-art on popular benchmarks in several language tasks.
However, such models are susceptible to adversarial attacks (Wang et al.) and reduced accuracy on out-of-distribution data (Du et al., 2021).
Bias and toxicity evaluations of such models are now standard with a variety of benchmarks (Ousidhoum et al., 2021).
Inspired by both the neurolinguistic ﬁndings about the curious case of personal names in human cognition and the now well-established bias studies in deep learning, we study personal names as a new source of variations in the performance of large language models even when the underlying reasoning task ought to be independent of the choice of personal names.
We create a scalable spatial reasoning task involving n individuals with different names.
In this task, we provide the pairwise spatial relationship between entities to the BLOOM family of models, and then ask the model to predict a new relationship among the entities.
Using male names popular in the US on the spatial reasoning task with four individuals, the accuracy of the BLOOM model grows from 0.3 to an impressive 0.83 as the number of parameters increases from 560 million tobillion.
Thus, an increase in the model size leads to an improvement in the accuracy of the model’s accuracy in reasoning about the spatial relationships, and sufﬁciently large models exhibit very good spatial reasoning capability.
Variation of accuracy based on names: The accuracy of the BLOOM models on this spatial reasoning task varies signiﬁcantly depending on the source countries of the proper names even though the reasoning task is completely invariant to the choice of the names.
The difference in accuracies for popular proper names from USA, Morocco, South Africa and China are shown in Fig 2(bottom right).
When we change the names in the reasoning task using the names common in US, the variance is
but it increases to 0.08 for names from Morocco, 0.12 for South Africa, and 0.24 for China.
Further, the difference in accuracies depending on the geographical region of the proper names is generally exacerbated in larger BLOOM models.
Poor calibration of BLOOM models on spatial reasoning :
We observe that the conditional log probabilities or beam scores of BLOOM predictions are not well-calibrated, and the scores do not discriminate between correct and wrong responses.
The beam scores of 100 responses from
BLOOM-176B are shown in Fig.
2 (top right).
The scores for the correct (incorrect) responses are shown in blue (red).
As shown in the ﬁgure, the distributions of the two scores can not be separated.
Thus, this approach for uncertainty quantiﬁcation is not sufﬁcient to detect when the model makes wrong predictions due to name bias.
We believe that our results show that there is an urgent need to develop training strategies and inference algorithms that can mitigate emergent bias for large language models in such settings, where performance in the underlying task ought to be clearly independent of the source of the bias.
Spatial Reasoning in BLOOM Large Language Models 2.1 Spatial Reasoning Tasks
[Table]
Spatial reasoning has been a topic of sustained interest in representational learning.
Qualitative reasoning using spatial cardinal directional representations has been investigated for at least three decades (Frank, 1991; Teresa Escrig & Toledo, 1998).
Spatial reasoning for textual data (Weston et al., 2015) using deep learning models (Le et al., 2020) has made such rapid progress that has led to the need for creating new and more challenging benchmarks (Shi et al.).
We create a simple spatial reasoning task where n individuals are located on a straight line.
The task speciﬁes the relative location, left or right, of one person with respect to another.
Finally, the task requires that we make a new hitherto undeclared inference about the relative position of individuals.
We keep the reasoning problem simple because our goal is not to evaluate the limits of the reasoning capability of these large language models, but instead to analyze their bias with respect to the used proper names in deﬁning the reasoning problem.
The challenge problems in our tasks are created algorithmically.
Given n individuals and a query pair p k , p l , we construct a left-graph
G l =
( V, E l ) of n nodes.
For every pair of nodes p i , p j 2 V , we add the directed edge ( p i , p j ) to the left-graph if p i is known to be on the left of p j according to the stated facts in the generated task.
Finally, we compute the transitive closure G ⇤ l
= ( V, E ⇤ l ) of the left-graph.
If ( p k , p l ) 2 E ⇤ l , the given facts in the generated task are adequate to show that p k is to left of p l , we keep this problem in our task and note that the correct response states that p k is to left of p l .
Similarly, we create a right-graph where the edges denote that an individual is to the right of another, compute its transitive closure and determine whether the stated facts are enough to conclude that p k is to the right of p l .
An illustrative automatically generated task is shown below with the correct response from the BLOOM-176B model colored in red.
John is on the right of David.
James is on the left of David.
James is on the left of John.
Joseph is on the right of James.
John is on the left of Joseph.
Then, we can make a new inference that David is on the left of Joseph.
2.2 BLOOM Large Language Models
The recent increase in the size of large language models using distributed GPUs has led to impressive performance on diverse tasks, such as ﬁnishing sentences (Zellers et al., 2019), commonsense story cloze (Mostafazadeh et al., 2016), physical commonsense reasoning (Bisk et al., 2020), challenging question answering problems (Clark et al., 2018), open book question answering (Mihaylov et al., 2018), Turing tests based on correct word disambiguation (Levesque et al., 2012; Sakaguchi et al., 2021), and more challenging general-purpose language understanding (Wang et al.).
A variety of large language models have been trained on internet-scale corpora of text in multiple languages and code bases.
These models include BERT, T5 , GPT, OPT, PALM and BLOOM.
A few of these models, such as GPT3, are only accessible via an API while all the trained weights for other models, such as BLOOM, are available to the public at large.
Our experimental studies have focused on BLOOM as it is an open-science open-access model that has been trained using data in multiple languages and is the result of an international effort.
BLOOM has been trained on the Jean Zay Public Supercomputer provided by the French government and is readily available to the public (BigScience, 2022).
BLOOM is a family of large-language models ranging from 560 million parameters to 176 billion parameters, and provides an effective platform for evaluating the variations in our spatial reasoning task.
BLOOM models have been trained on substantial text from multiple languages and probably multiple countries.
2.3 Efﬁcacy of BLOOM on Linear Spatial Reasoning Task
We observe that BLOOM is capable of solving the spatial reasoning task with zero-shot prompting in many cases.
An example can be seen in the illustration in Fig.
We consider reasoning tasks with different complexities by changing the number of individuals over which spatial reasoning needs to be performed.
We performed experiments on the BLOOM family of large language models on a server with 256 AMD cores, 2 TB of RAM and 8 Nvidia A100 80GB GPUs.
Each language task was performed 100 times to compute the average accuracy of the task.
Our list of popular names in different countries was ﬁrst obtained from https://en.wiktionary.org/wiki/Appendix:
Most_popular_given_names_by_country .
This list was supplemented by the names available at https://en.wikipedia.org/wiki/List_of_most_popular_given_names .
[Figure]
Spatial reasoning over 3 individuals. Figure 3 shows the performance of the BLOOM family of models on our spatial reasoning task for 3 male names drawn from the popular names of different countries.
The accuracy of the BLOOM family rsies in a sustained manner from a maximum of 0 .
52 for the 560 million parameter model to a maximum of 1 . 0 for the 176 billion parameter model.
[Figure]
Spatial reasoning over 4 individuals.
Figure 4 shows the performance of BLOOM family of models on the spatial task with 4 male names drawn from the popular names of different countries.
The 560 million parameter model has a maximum accuracy of only 0 . 3 , which is below random chance.
However, thebillion parameter model has an accuracy of 0 .
83 for the United States.
Accuracy Variations due to Country of Personal Names
While the signiﬁcant variation of the model accuracy on the geographical locations of the names on an unrelated spatial reasoning task is itself a source of concern, we observe that the variation is emergent and it generally increases in magnitude as the models grow in size.
We quantitatively evaluate the variation as the difference between the accuracy for a given geographical area and the maximum accuracy for a model of that size.
For example, for 4 individuals, names drawn from
China have an accuracy of 0.59 while names drawn from USA have an accuracy of 0.83 for the
BLOOM-176B model.
Hence, the variation for China is the difference of the two accuracies, i.e., 0.24.
This high variation indicates relatively low accuracy for the corresponding subpopulation even when the underlying reasoning task is the same and should not depend on the names being used, and hence, indicates bias in the model.
Fig 5 shows a plot of the variation for multiple BLOOM models with four names drawn from popular names in different countries.
For names popular in USA, a rise in the size of the model reduces the variation to 0.
On the other hand, a rise in the size of the BLOOM models generally exacerbates the variation in accuracy problem for countries, such as China, India, and South Africa.
[Figure]
We also analyze the variation on the more challenging spatial reasoning task with 5 individuals.
Figure 6 shows that the BLOOM-176B model shows a variation of 0.03 for the United States.
However, the variation for China, India, and South Africa becomes as high as 0 .
[Figure]
i n A cc u r ac y 560M 3B
7B
176B
These variations may be explained using various reasons: (i) different term-frequency in training data, (ii) lack of descriptive value of a personal name that can be learned using a large text corpora, and (iii) the potentially larger set of possible phonologies in different languages.
However, as classical algorithms are replaced by increasingly complex and often proprietary and closed large language models, the variation from proper names needs to be addressed to avoid biased analyses.
Uncalibrated Model Scores for Predictions
Since the BLOOM models produce signiﬁcantly different accuracies on the spatial reasoning task for different choice of names, it becomes even more important to understand if the model is predicting a correct response on a given query.
A natural approach to quantify uncertainty of a model is to use the log probability of the response predicted by the model and threshold it to decide if a response is reliable or not.
We show that the BLOOM models are not well-calibrated and produce similar conditional log probabilities for both correct and incorrect responses.
The BLOOM models, like other large language models, compute the beam score or the conditional log probability of the response being predicted.
In a well-calibrated model, a correct output will be associated with a high probability and an incorrect output will correspond to a relatively lower probability.
This will allow the model to be deployed in a practical setting.
[Figure]
Fig 7 shows the beam score or log probability corresponding to a typical response from the BLOOM176B model on male names from the US, China and South Africa.
The horizontal axis represents the index of the query and the vertical axis represents the beam score or the log probability of the response.
Conclusions and Future Work
While there has been signiﬁcant work on evaluating explicit bias in the predictions from large language models, we propose the automated synthesis of a spatial reasoning task that ought to be independent of the choice of personal names.
However, we ﬁnd that the BLOOM large language models are substantially inﬂuenced by the choice of personal names on this task.
Further, the beam score or conditional log probabilities of predicted tokens have very poor correlation with the correctness of the model response on this task, and cannot be relied upon as a metric for uncertainty quantiﬁcation.
The analysis and observations in this paper open up a couple of exciting directions for future research.
First, it will be interesting to compare the performance of BLOOM with other large language models that were not designed by an international team of researchers.
Since the trained weights of many of the state-of-the-art models are not available to the public at large, such proprietary models need similar internal auditing.
Second, our work has focused on personal names, and it may be interesting to pursue a wider study using different proper nouns, such as places and brand names.
Ethics Statement
Our approach brings to light a potential concern with BLOOM and possibly other large language models where their accuracy is affected by the choice of personal names, even on tasks that are completely independent of the choice of names.
By increasing the awareness of potential bias of these large language models, we hope to motivate research into mitigation techniques and make practitioners aware about challenges in social deployment of these models.