Fair Text-Attributed Graph Representation Learning (2025)

Abstract: Text-Attributed Graphs (TAGs), which integrate text and graph structures, have recently gained traction, especially in web applications. However, as a graph structure, TAG representation learning (TAGRL) naturally inherits issues from Graph Neural Networks (GNNs), such as fairness. Moreover, previous TAGRL research has mainly focused on using LM-as-encoder to boost downstream task performance, with little consideration given to whether this process may raise additional concerns related to fairness and other safety-related issues. As the first work to explore fairness in TAGRL, this paper proposes the concept of evolving LM-as-encoder to LM-as-fair-encoder, developing a two-stage fairness-aware alignment process called FairTAG based on the observed issues. Specifically, we first mitigate the tendency of LMs to overfit to homophily during downstream tasks fine-tuning, followed by subgraph-level connection behavior preference optimization for selected anchor nodes. We provide theoretical support and demonstrate the feasibility of LM-as-fair-encoder through extensive experiments and ablation studies. We also show that FairTAG can be seamlessly integrated with fairness-enhancing strategies on the GNNs decoder side, thus innovatively constructing a plug-and-play learning framework.

## introduction text-attributed graphs (tags) are a prevalent data structure where nodes are enriched with textual information, combining structured and textual knowledge @xcite @xcite . widely used in applications like recommendation systems and information retrieval @xcite @xcite , these graphs are increasingly central to decisionmaking systems. however, like their traditional counterparts, tags inherit significant fairness challenges from graph neural networks (gnns) @xcite @xcite . ensuring that models trained on tags do not perpetuate or amplify biases across demographic groups is a critical, yet underexplored, problem. the dominant paradigm for text-attributed graph representation learning (tagrl) is lmas-encoder @xcite @xcite , where a language model (lm) is fine-tuned to generate rich node embeddings. as shown in figure 1 , this paradigm has successfully improved task performance (e.g., auc). however, our preliminary analysis, using the well-established mixed-dyadic demographic parity (dp m ) metric @xcite , reveals a troubling trend: state-of-the-art lm-as-encoder methods often exhibit fairness scores equal to or worse than simpler methods, particularly on social network data. this raises a crucial question: are lms, despite their power, inadvertently amplifying biases present in the graph? the core intuition, which underpins our work, is that this fairness degradation is a direct consequence of how lms learn on biased graph data. real-world graphs are often dominated by homophily, where intra-group links vastly outnumber inter-group links. when fine-tuned, lms tend to overfit to this majority pattern. they can achieve low training loss by simply learning this "easy" homophilous pattern, while failing to correctly model the "harder," less frequent, but crucial inter-group connections. this creates a significant fairness problem. to address this, we propose advancing the paradigm from lm-as-encoder to lm-as-fairencoder and introduce fairtag, a two-stage framework designed to achieve both effectiveness and fairness. our approach is built on a clear logical progression from a coarse-grained global correction to a targeted, local refinement. stage 1: global debiasing with fairness score-based edge drop (fsed). our first step is to combat the model's tendency to overfit to "easy" samples. we define simple samples as edges a model predicts correctly with high confidence (overwhelmingly the majority-class, intra-group links) and difficult samples as those it misclassifies or predicts with low confidence (disproportionately the minority-class, inter-group links). to systematically identify these, we first train an oracle model on the original biased data to serve as a stable proxy for the difficulty inherent in the dataset. fsed then uses the discrepancy between the oracle's predictions and the ground truth to assign a difficulty score to each edge. by probabilistically dropping the simplest, high-confidence homophilous edges, fsed effectively re-balances the training data's difficulty. this forces the subsequent model to learn from a more challenging dataset, compelling it to grasp the nuanced patterns of inter-group connections rather than relying on the easy homophily shortcut. stage 2: local refinement with anchor node preference optimization (anpo). while fsed provides a crucial global correction, it treats all nodes monolithically. however, a node deep within a homophilous cluster faces different challenges than one at the boundary between groups. to address this limitation, we introduce a targeted, local intervention. we identify anchor nodes as nodes at the boundaries of sensitive attribute clusters, which have a more balanced mix of intra-and inter-group connections and are most vulnerable to fairness violations. for these critical nodes, we use anpo, a preference optimization algorithm inspired by dpo @xcite , to explicitly teach the model to value inter-group connections more eq-uitably. we frame this as a preference task: for a given anchor node, a valid inter-group connection is preferred over a valid intra-group connection. this allows us to surgically target the local fairness gap with a level of precision that a global strategy cannot achieve. our main contributions are summarized as follows: ## methodology in this section, we introduce our fairness-aware lm training paradigm, fairtag, which is the first framework to focus on fair lm training on tags. this pipeline can be seamlessly integrated with existing gnn architectures and connected with fairness research in the gnn domain, demonstrating its versatility. we propose a two-stage alignment method to mitigate unfairness at the embedding level before applying gnns. this includes: i) a fairness-score based edge drop strategy that reduces the learning inertia of lms on both interclass and intra-class edges; and ii) a subgraph-level preference optimization to enhance local fairness in predictions. newly defined symbols, as well as those from section 3, are used in this section. a detailed summary of the notation is provided in appendix h. theorem 4.1. when considering a k-layer gnn, if the following assumptions hold: for any node pair s @xmath0, the features processed by language model f(•) result in a reduced distance between nodes with the same sensitive attribute, that is: the proof is in appendix b.1. additionally, we conduct an analysis to measure the average squared euclidean distance between intra-class node pairs after different encoding stages. the results in ## compatibility with gnn-side methods to bridge the gap with fairness-aware methods on the gnn decoder side and establish fairtag as a plug-and-play approach, we demonstrate its compatibility. specifically, fairtag, when trained with gnns using fairdrop or dropedge, exhibits superior performance for ∆dp m (figure 3 ). this indicates fairtag, pioneering fairness optimization on the lm side, is compatible with decoder-side techniques, enabling a cascaded optimization pipeline. we also show that individual use of fairdrop or dropedge does not match the performance of joint training. this establishes fairtag as an effective plug-and-play method in tagrl. ## conclusion our work pioneers fairness in lm and gnn collaboration on tags. we identify fairness concerns in lm-as-encoder architectures and demonstrate that lm-specific techniques can mitigate bias during feature embedding. our proposed two-stage alignment method, fairtag, offers an innovative plug-and-play approach compatible with existing fair gnn training methods. fairtag uniquely incorporates an edge drop based on the fairness score and an anchor node-based edge generation, enhancing fairness at both the global and subgraph levels. empirically, fairtag surpasses current lm-based and leading gnn decoder side techniques in link prediction accuracy and fairness. theoretically, we highlight that the lm-induced bias can be amplified by gnns; our method mitigates this by reducing the lm encoding bias and correcting the lower bound of the amplified bias. this research underscores the potential of re-envisioning lms for fairness, paving the way for more equitable ai systems on text-rich graph data.