Model-Agnostic Bias Measurement in Link Prediction (2023)

Abstract: Link prediction models based on factual knowledge graphs are commonly used in applications such as search and question answering. However, work investigating social bias in these models has been limited. Previous work focused on knowledge graph embeddings, so more recent classes of models achieving superior results by fine-tuning Transformers have not yet been investigated. We therefore present a model-agnostic approach for bias measurement leveraging fairness metrics to compare bias in knowledge graph embedding-based predictions (KG only) with models that use pre-trained, Transformer-based language models (KG+LM). We further create a dataset to measure gender bias in occupation predictions and assess whether the KG+LM models are more or less biased than KG only models. We find that gender bias tends to be higher for the KG+LM models and analyze potential connections to the accuracy of the models and the data bias inherent in our dataset. Finally, we discuss the limitations and ethical considerations of our work. The repository containing the source code and the data set is publicly available at https://github.com/lena-schwert/comparing-bias-in-KG-models .

## introduction achieving reliable link prediction in factual knowledge graphs (kgs) is an important goal to overcome the inherent gaps in their knowledge. such graphs are widely used by companies such as google, linkedin, amazon, and bloomberg across a range of different real-world applications, including search, recommender systems, and voice-based question answering @xcite @xcite . typically, information is stored in the shape of triples (h, r, t), consisting of a head entity h, a relation r, and a tail entity t. entities express concepts, while relations express the connection between them, e.g., @xcite . link prediction models score the plausibility of a given fact, with two distinct purposes: (i) they make the graph structure available to machine learning models, e.g., in the form of embeddings, (ii) and -if sufficiently reliable -may eventually be used to make plausible predictions of missing facts, i.e., solving the problem of automatic knowledge graph completion and refinement @xcite . while link prediction models are naturally evaluated for their accuracy, there have only recently been studies that assess possible biases that they may exhibit. echoing prior position papers on bias in factual kgs @xcite , we consider an analysis of bias as essential for a thorough model evaluation, especially because a) kgs contain sensitive information about humans (e.g., gender), b) historical facts naturally contain historical biases, and c) the applications of kg-based models are increasingly socially relevant due to their proliferation into widely deployed systems such as search engines and conversational agents @xcite . to achieve a meaningful bias analysis for the link prediction task, we argue that a modelagnostic approach is necessary. only then can bias be measured comparatively across different model classes, highlighting strengths and weaknesses as well as potential causes for biased behavior. however, existing bias measurement approaches are highly model-dependent (for a recent in-depth review, refer to kraft and usbeck 2022). they focus only on knowledge graph embeddings (kges) @xcite @xcite @xcite @xcite , the earliest class of neural link prediction methods, which approximate an existing kg by exploiting the structural information contained in the facts of a kg @xcite . as kgs are incomplete and typically contain a large of number of entities that only appear in a few triples, more recent text-based models incorporate additional textual data sources for improved results @xcite . pre-trained language models (lms) based on transformers @xcite have successfully been shown to achieve this @xcite @xcite , significantly improving the accuracy on benchmark datasets. however, models of this sort have not yet been investigated for bias. we thus propose to conduct model-agnostic evaluations of link prediction models, enabling us to compare bias between representatives of kge models and lm-based link prediction models, which we henceforth refer to as kg only and kg+lm models, respectively. however, we stress that numerous other link prediction model classes exist that our approach can be used for @xcite . like previous work by @xcite , our notion of bias ( §2) draws on group fairness metrics for classification tasks (as reviewed by @xcite . these metrics are extrinsic metrics @xcite , meaning that they measure performance differences on a specific downstream task, i.e., link prediction, for different social groups (e.g., gender). our paper makes the following contributions: ## bias statement and definitions we follow blodgett et al. (2020) who stress the importance of making the authors' understanding of bias explicit whenever it is investigated, following the taxonomy of harms @xcite . our understanding of bias in link prediction is based on the idea of representational harm, more specifically, "differences in system performance for different social groups" @xcite . for example, in the case of gender bias in occupation predictions, a link prediction model that predicts the occupations of women less accurately than those of men would be deemed as behaving harmfully. we consider this behavior as harmful, because deploying such a biased model in a downstream application can make it less useful for women than men. the extent of harm that is caused depends on the societal relevance of the application, e.g., it might be used for job applications or credit approval. beyond measuring bias in link prediction model predictions, we also investigate data bias in the knowledge graph (kg) datasets that we use. we consider the data to be biased if it is highly imbalanced with regard to some ideal distribution across social groups. an example of such an imbalance would be a dataset that contained significantly more facts about men than women and no facts about individuals with other gender identities. the examples above also show that we measure bias in a specific context, defined by a sensitive attribute and a target property. a sensitive attribute is an inherent characteristic of an entity worthy of (legal or other) protection. typical examples are gender, race, ethnicity, religion or worldview, disability, age, and sexual identity. each sensitive attribute usually defines multiple groups, categorical options that a person can belong to, e.g., female gender. we view a target property as some notable property or achievement of an entity. typical examples are their occupation, awards received, degrees, or where a person was educated. both the target property and the sensitive attribute need to be expressed by one specific relation in the dataset. this means that the dataset-specific meaning of the respective relation matters: here, we want to measure the influence of gender on the accuracy of occupation predictions. following @xcite , we define gender as a multiplicitous concept expressing, e.g., identity and behaviors, going beyond bodily attributes that determine the biological sex of a person. while we only discuss gender in the following, we note that for the gender identities "female" and "male" that we analyze, the distinction between gender and sex is not explicit in wikidata, because the same entity is used to express both concepts. 1 in addition, due to data scarcity, we cannot analyze gender bias for other gender identities, such as "non-binary". foot_1 due to a lack of reliable information, we do not distinguish between cisgender and transgender individuals when performing our analysis of gender bias for women and men. we refer to the hci gender guidelines for further information about the terminology and concepts discussed above. 3 ## conclusion we present a model-agnostic approach for measuring bias in link prediction along with the first experimental study that measures bias in language model (lm)-based link prediction models (kg+lm), comparing it with bias in knowledge graph embedding (kge) models (kg only). using a selection of fairness metrics and analyzing our results at three levels of detail, we find that the kg+lm models are more biased. we discuss the relationship between bias, link prediction accuracy metrics and data bias. for our experiments, we create humanw5m-3mil, a subset of 3 million facts about humans contained in wikidata @xcite . we have made our code and the dataset available to the public to encourage further research on these topics.