H yper BERT : Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs (2024)

Abstract: Hypergraphs are characterized by complex topological structure, representing higher-order interactions among multiple entities through hyperedges. Lately, hypergraph-based deep learning methods to learn informative data representations for the problem of node classification on text-attributed hypergraphs have garnered increasing research attention. However, existing methods struggle to simultaneously capture the full extent of hypergraph structural information and the rich linguistic attributes inherent in the nodes attributes, which largely hampers their effectiveness and generalizability. To overcome these challenges, we explore ways to further augment a pretrained BERT model with specialized hypergraph-aware layers for the task of node classification. Such layers introduce higher-order structural inductive bias into the language model, thus improving the model’s capacity to harness both higher-order context information from the hypergraph structure and semantic information present in text. In this paper, we propose a new architecture, HyperBERT, a mixed text-hypergraph model which simultaneously models hypergraph relational structure while maintaining the high-quality text encoding capabilities of a pre-trained BERT. Notably, HyperBERT presents results that achieve a new state-of-the-art on five challenging text-attributed hypergraph node classification benchmarks.

## introduction in recent years, the study of hypergraphs has gained significant attention in the field of network analysis @xcite @xcite . unlike traditional graphs, where edges represent pairwise relationships, hypergraphs are a generalization of graphs where an edge can join any number of nodes, allowing for more complex interactions by connecting multiple nodes through hyperedges. this characteristic makes hypergraphs an ideal framework for representing complex systems such as social networks, biological networks, and collaboration networks, where relationships often involve sets of entities. machine learning on hypergraphs @xcite , has been shown to be an effective tool for studying complex graph-based data structures. in particular, when compared with a standard graph, a hypergraph is more capable and flexible in modeling high-order relations since one hyperedge in a hypergraph can connect more than two nodes. as a result, hypergraph learning has recently gained increasing attention and has been applied to many research fields, such as computer vision @xcite and recommendation systems @xcite . in many real-world complex systems, hypergraph data is accompanied by node attribute information @xcite @xcite . some examples of node attributes include the age, ethnicity, and gender of individuals in social networks @xcite , affiliations of authors in collaboration networks @xcite , and article abstract texts in co-citation graphs @xcite . previous studies have demonstrated that node attribute data potentially enhances the learning of community structure in networks @xcite . a particular instance of attributed hypergraphs are text-attributed hypergraphs (tahgs) @xcite @xcite , which have been widely used for modeling a variety of realworld applications, such as co-authoring in collaboration networks @xcite , information retrieval @xcite , product recommendation @xcite , and many others. given the prevalence of tahgs @xcite @xcite , we have explored how to effectively handle these graphs, with a fo-cus on node classification tasks. such tasks aims at predicting the category or label of a node based on its textual attributes and the structure of the hypergraph. this problem is pivotal in various applications such as predicting protein functions in biological networks @xcite , identifying roles in social networks @xcite , and classifying documents in citation networks @xcite . inherently, tahgs provide both node attribute and graph structural information. thus, it is critical to effectively capture both while modeling their interrelated correlation for effective representation learning. node classification in text-attributed hypergraphs presents unique challenges that are not encountered in traditional graph-based approaches @xcite @xcite . in this context, there are two key hurdles. first, since hyperedges can connect any set of nodes, capturing such complex interaction patterns becomes increasingly difficult using conventional graph-based methods. second, effectively exploiting the textual information to learn informative node representations is complex. recent advances in the field of nlp have led to the emergence of language models (lms) with strong capabilities for semantic understanding of text @xcite @xcite @xcite . however, the integration of hypergraph structural information directly into language models is still an open and under-researched topic. to address the above limitations, we propose a novel architecture, hyperbert, which mixes hypergraph-aware layers into language models to simultaneously exploit hypergraph topology and textual representations for the node classification task on text-attributed hypergraphs. in particular, hyperbert extends the original bert architecture with hypergraph-specific inductive biases. to do so, we design specially-crafted hypergraph-aware layers into the bert encoder, effectively mixing hypergraph structure with text semantic embeddings for computing node representations. to accomplish the text-hypergraph alignment in the feature space, we propose a novel self-supervised loss for pretraining hyperbert and effectively aligning semantic and hypergraph feature spaces. hyperbert achieves state-ofthe-art performance across five widely used hypergraph benchmarks. our code is available at https://github.com/adrianbzg/hyperbert . ## related work our work is related to two lines of research: textattributed graph learning-based methods and language model-based approaches on text-attributed graphs @xcite @xcite . in this section, we review briefly the relevant literature for such categories. given the recent success of graph representation learning, numerous research works have been proposed for various tasks including node classification @xcite or link prediction @xcite . graph neural networks (gnns) are recognized as the de facto standard technique for modeling graph data. these methods (e.g. gcn; @xcite , gat; @xcite , graphsage; @xcite or gin; @xcite ) learn effective mechanisms such that information between the nodes is aggregated for expressive graph representations. in the case of tags, a cascade architecture @xcite is typically adopted. in such setting node features are encoded independently using text modeling techniques (e.g. bag-of-words @xcite , skip-gram @xcite , or more recently language models such as bert @xcite ), and subsequently aggregated via the message passing mechanism to produce the final node representations. however, standard gnn-based approaches are only applicable to general graphs with binary relations. to alleviate this issue recent research has focused on developing effective methodologies for attributed hypergraph learning in order to capture nary or high-order interactions between nodes @xcite @xcite . for instance, @xcite proposes an inductive multi-hypergraph learning algorithm, which learns an optimal hypergraph embedding with good performance on the task of multi-view 3d object classification. more recently, several hypergraph-based methods have been proposed for the task of node classification @xcite . in particular, @xcite propose the hypergraph neural network (hgnn) architecture, which encodes highorder node correlations using a hypergraph structure and generalizes the convolution operations to the hypergraph learning process based on hypergraph laplacian and truncated chebyshev polynomials. @xcite propose to supplement the family of graph neural networks with two end-to-end trainable operations, i.e. hypergraph convolution and hypergraph attention (hcha). @xcite propose hypergcn, model to enhance the hypergraph laplacian with additional weighted pair-wise edges. @xcite propose hype-boy, a hypergraph self-supervised learning method for node classification. however, hypeboy primarily focuses on hypergraphs without specifically addressing the complexities introduced by nodes with textual attributes. given the success of lms in solving semantic representation learning tasks, recent works @xcite have proposed to utilise lms for tags, as they can transform the nodes' textual attributes into semantically rich node embeddings. however, node embeddings provided by lms often only contain information from the textual attributes, overlooking the topological structure present in the tags. to overcome this limitation, multiple works integrate lm-based embeddings with graph learning methods @xcite , with the goal of generating node embeddings that are tailored for specific tag tasks. for instance, @xcite propose graphormer, which fuses lm-based text encoding and graph aggregation into an iterative workflow. giant @xcite introduces a novel neighborhood prediction task to fine-tune a xr-transformer @xcite , and feeds node embeddings as obtained by the lm into gnns for downstream tasks. tape @xcite presents a method that uses lms to generate prediction text and explanation text, which serve as augmented text data to subsequently feed into gnns. lm- @xcite introduces graphaware pre-training to adapt the lm on the given graph before fine-tuning the entire model, demonstrating significant performance results. simteg @xcite finds that first training the lms on the downstream task and then freezing the lms and training the gnns can result in improved performance. k-bert @xcite leverages a knowledge layer to inject relevant triplets from a knowledge graph into the input sentence and transform it into a knowledge-rich sentence tree. kg-bert @xcite treats triples in knowledge graphs as textual sequences. they propose to model entity and relation descriptions of a triple as input, and compute the scoring function of the triple with the kg-bert language model. despite these advances, previous works do not take into account the problem where both text attributes and higher-order interactions exist between the entities, as is the case with text-attributed hypergraph node classification. ## preliminaries in this paper, we focus on learning representations for nodes in text-attributed hypergraphs (tahg), where we take node classification as the downstream task. therefore, before describing the details of our proposed method, we start with presenting a few basic concepts, including the formal definition of tahgs and how language models can be used for node classification in tahgs. then, we introduce the formulation of the problem our method is tackling. @xcite where nodes are connected between each other using binary relations, and a text-attributed hypergraph (right) where nodes are related with high-order connections (hyperedges). in both cases, each node is attributed with a textual description, such as paper abstract in the case for co-citation hypergraphs. ## methodology in this section we present our proposed model, hy-perbert, as illustrated in figure 2 . first, we begin with the details of the hyperbert layer design. then, we detail our proposed pre-training methodology, followed by details of the different components of our knowledge alignment loss function. finally, we describe how fine-tuning is performed for a downstream node classification task. ## experiments in this section we show our model performance on five hypergraph node classification benchmarks from a variety of domains. also, we present ablation studies to analyse the importance of the different components of the hyperbert model. ## conclusion in this work, we propose a new model architecture to improve the capability of hypergraph structural encoding of bert effectively while retaining its semantic encoding abilities. in order to achieve this, we designed hyperbert, a hypergraph-aware language model based on bert that augments with structural context for the challenging task of textattributed hypergraph node classification. notably, extensive experiments on five benchmark hypergraph datasets demonstrate the effectiveness of hy-perbert for text-attributed hypergraph node classification tasks, proving that simultaneously modelling hypergraph structure and semantic context is crucial to achieve state-of-the-art results.