Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors (2025)

Abstract: We propose Fast-and-Frugal Text-Graph (FnF-TG) Transformers, a Transformer-based framework that unifies textual and structural information for inductive link prediction in text-attributed knowledge graphs. We demonstrate that, by effectively encoding ego-graphs (1-hop neighbourhoods), we can reduce the reliance on resource-intensive textual encoders. This makes the model both fast at training and inference time, as well as frugal in terms of cost. We perform a comprehensive evaluation on three popular datasets and show that FnF-TG can achieve superior performance compared to previous state-of-the-art methods. We also extend inductive learning to a fully inductive setting, where relations don’t rely on transductive (fixed) representations, as in previous work, but are a function of their textual description. Additionally, we introduce new variants of existing datasets, specifically designed to test the performance of models on unseen relations at inference time, thus offering a new test-bench for fully inductive link prediction.

## introduction knowledge graphs (kgs) represent complex information as a structured collection of entities and their relations. they are a fundamental component of various applications, including information extraction @xcite @xcite and retrieval @xcite , question answering @xcite @xcite , reasoning @xcite @xcite , fact-aware language modelling @xcite , and many others @xcite ). text-attributed kgs extend kgs by associating each entity and relation with a corresponding textual description, which provide a richer representation of the knowledge encoded in the graph. in particular, the text associated with an entity may provide a description of its relationships to other entities. this combination of explicit structural and implicit textual information makes modelling text-attributed kgs particularly challenging. initial attempts to model kgs focused on their graph nature, typically addressing a transductive setting @xcite @xcite . these models could only make predictions for entities observed during training and only considered the structural information of the kg, ignoring any textual information. to overcome this limitation, later work focused on using the textual descriptions in kgs to address an inductive setting @xcite @xcite , meaning that predictions can be made even for entities not observed during training, using entity representations computed based on their textual descriptions. combining information from textual descriptions and graph structures has proven crucial @xcite . an entity's ego-graph, which represents it's 1-hop neighbourhood, provides valuable context that can help disambiguate its role and distinguish it from similar entities. while there has been progress in leveraging egographs, we believe that there is significant room for more effective approaches. modelling text-attributed kgs in an inductive setting poses several challenges, particularly when it comes to effectively integrating textual and structural information in embeddings. transformers @xcite have shown remarkable success at modelling unstructured (text) data @xcite @xcite . while their ability to model structured (graph) data is less evident, the inherent graph processing abilities of a transformer's self-attention mechanism make it a natural fit for modelling graph structures @xcite . we leverage recent advances in using transformers for graph encoding @xcite @xcite , and propose fast-and-frugal text-graph (fnf-tg) transformers, which unify textual and structural information in a framework based solely on transformers. another challenge is that text encoders are resource-intensive, especially when the textual descriptions of both entities and their ego-graph neighbours need to be encoded, leading to considerably increased training and inference time @xcite . this cost can be reduced by using smaller text encoders, but they can be considerably less effective. we demonstrate that we can reduce the dependence on large text encoders with a more effectively encoding of ego-graphs, using our fnf-tg transformers and their more appropriate inductive biases. this makes the overall framework both fast in terms of time, as well as frugal in terms of cost. a third challenge is that previous models fail to leverage the textual descriptions of relation labels. they still assume a fixed (transductive) inventory of relations, meaning that they cannot handle relations which they did not see during training and thus are not fully inductive. we propose an extension of this method to address the challenge of being fully inductive, by computing a relation embedding from the text describing that relation. this embedding serves as both the relation representation for link prediction, analogous to the transductive case, and also as the relation label which is input to the self-attention mechanism of the fnf-tg's graph encoder component. we showcase the effectiveness of our proposed model on three popular datasets for inductive link prediction in text-attributed kgs from the experimental setting of @xcite and @xcite , namely wn18rr ind , fb15k-237 ind , and wikidata-5m ind . we show that it improves over the state-of-the-art in all cases. additionally, we introduce new variants of existing datasets which are specifically designed to evaluate the performance of models on relations which are unseen until test time, thus offering a new test-bench for fully inductive link prediction. ## conclusion we presented a new transformers-based approach to link prediction in text-attributed knowledge graphs that combines textual descriptions and graph structure in a fully inductive setting. our fast-and-frugal text-graph (fnf-tg) transformers outperform previous state-of-the-art models on three popular datasets, showcasing the importance of capturing rich structured information about entities and their relations. our approach achieves superior performance while maintaining efficiency and scalability, making it a promising solution for large-scale knowledge graph applications. moreover, our ablation studies provide insights into the key factors contributing to its effectiveness, demonstrating the value of each component in our model. additionally, we proposed a new evaluation setting for fully inductive link prediction, where relations are also inductive, and demonstrated the potential of our approach in this setting.