VISP ool: Enhancing Transformer Encoders with Vector Visibility Graph Neural Networks (2024)

Abstract: The emergence of transformers has revolutionized natural language processing (NLP), as evidenced in various NLP tasks. While graph neural networks (GNNs) show recent promise in NLP, they are not standalone replacements for transformers. Rather, recent research explores combining transformers and GNNs. Existing GNN-based approaches rely on static graph construction methods requiring excessive text processing, and most of them are not scalable with the increasing document and word counts. We address these limitations by proposing a novel dynamic graph construction method for text documents based on vector visibility graphs (VVGs) generated from transformer output. Then, we introduce visibility pooler (VISPool), a scalable model architecture that seamlessly integrates VVG convolutional networks into transformer pipelines. We evaluate the proposed model on the General Language Understanding Evaluation (GLUE) benchmark datasets. VISPool outperforms the baselines with less trainable parameters, demonstrating the viability of the visibility-based graph construction method for enhancing transformers with GNNs.

## introduction the advent of transformer-based models has significantly propelled recent advancements in natural language processing (nlp), particularly in natural language understanding (nlu). transformers, introduced by @xcite , paved the way for models like gpt @xcite and bert @xcite and their variants. due to their ability to capture long-range dependencies and contextual information, these models outperform previous methods in various nlu tasks. in parallel, graph neural networks (gnns) have been introduced as potent tools for processing struc-tured data, including graphs in nlp tasks. the gnns can effectively capture underlying relationships and dependencies by leveraging messagepassing techniques @xcite . notably, @xcite introduced graph convolutional networks (gcns), which have been adapted for text-based graphs in models like textgcn @xcite and ten-sorgcn @xcite , along with other gnnbased methods in nlp @xcite @xcite @xcite . these methods have shown promise in various nlp tasks by effectively aggregating information from underlying graphs. however, most gnns work with a transductive structure that combines all training and test instances (nodes) in a single graph and makes predictions for the test nodes. therefore, adding new training instances or getting predictions on new test instances changes the graph structure, making the previously trained weights obsolete and requiring re-training of the network. hence, compared to transformer-based approaches, on their own, gnns do not perform as well in inductive learning settings, where the model is required to generalize to unseen data and continuously applied after training. the inherent transductive nature of gnns also raises an important caution by departing from the traditional assumption of completely unseen test data. these limitations have led to the development of (1) inductive gnns @xcite @xcite , (2) transductive methods that combine transformers with gnns @xcite @xcite , and finally (3) transformers with inductive gnns @xcite . transductive gnn methods are mostly built on a single large graph structure for the entire corpus with both training and test data, so they are neither applicable to unseen data nor scalable with the increasing document and word counts @xcite , 2019; @xcite . although the current inductive gnn methods construct individual graphs for each document, they rely on static graph construction methods based on statistical metrics such as term frequency-inverse document frequency (tf-idf), point-wise mutual information (pmi), and co-occurrences @xcite . these methods require extensive text preprocessing and are still prone to scalability issues. in a different vein, visibility graphs (vgs) have been introduced as a novel approach for mapping time series data into a graph @xcite @xcite . then, @xcite extends the notion to vector visibility graphs (vvgs) to map multivariate time series. the constructed graphs enable the extraction of sequential relations, facilitating improved analysis and insights. recent studies introduce gnns with vg @xcite , but to our knowledge, vvg-based gnn has not been proposed before. in a broader perspective, transformers generate document embeddings as multivariate time series with inherent sequential relations, where vvgs are designed to capture sequential relations. in contrast to statistical methods, vvg preserves sequential topology and can utilize transformer output without text processing. to leverage these capabilities, in this work, we propose (1) a novel approach to generate dynamic graphs per document by constructing vvgs from transformer representations and (2) a novel scalable architecture, visibility pooler (vispool), that seamlessly integrates vvg-based gcns into transformer pipelines and enhances performance on general language understanding evaluation (glue) benchmark @xcite . ## method we first briefly outline the mainstream approach and then introduce our novel model for enhancing transformer encoders with vvg-based gcns, where both architectures are shown in fig. 1 . ## experiments we evaluate our approach by addressing whether, given a pre-trained transformer, the vvg-based gcns can enhance the performance in nlu compared to the base classifier. due to their lightweight but competitive performance, we choose distil-bert @xcite and albert @xcite as base transformers and use their corresponding tokenizers, both provided by hugging face 2 . up to this point, the process is identical for the base classifier and vispool. then, for the base classifier, we use sequenceclassifier 2 head. on the other hand, for the vispool path, we implement a fast and parallel vvg generation process, along with a batched gcn layer implementation that extends the notion in pytorch geometric @xcite , to minimize the computational overhead, resulting in similar training times with 2 we use models from huggingface.co/distilbert/distilbertbase-uncased and huggingface.co/albert/albert-base-v2. the base classifier. further details of the experimental setup, including model details and computational infrastructure, are provided in appendix b. ## conclusion in this work, we demonstrate both the viability of the vvg approach for graph construction and the effectiveness of the proposed vispool architecture for enhancing transformer encoders with gnns in nlu tasks. vispool is shown to be successful in aggregating relational information that contextual transformers cannot capture. therefore, we believe that the vvg-based dynamic graph construction is a promising approach for nlp tasks thanks to not needing excessive text processing and being more scalable than existing approaches. the proposed vispool architecture can be further explored and extended by weighted vvgs and applications to other inductive nlp tasks that can benefit from transformer enhancements with gnns.