C on T ext ING : Granting Document-Wise Contextual Embeddings to Graph Neural Networks for Inductive Text Classification (2022)

Abstract: Graph neural networks (GNNs) have been recently applied in natural language processing. Various GNN research studies are proposed to learn node interactions within the local graph of each document that contains words, sentences, or topics for inductive text classification. However, most inductive GNNs that are built on a word graph generally take global word embeddings as node features, without referring to document-wise contextual information. Consequently, we find that BERT models can perform better than inductive GNNs. An intuitive follow-up approach is used to enrich GNNs with contextual embeddings from BERT, yet there is a lack of related research. In this work, we propose a simple yet effective unified model, coined ConTextING, with a joint training mechanism to learn from both document embeddings and contextual word interactions simultaneously. Our experiments show that ConTextING outperforms pure inductive GNNs and BERT-style models. The analyses also highlight the benefits of the sub-word graph and joint training with separated classifiers.

## introduction recently, the methods of non-sequential text modeling have gained attention, particularly for graph neural networks (gnns) that learn document representation from graph structures. most gnns @xcite @xcite are transductive since they are designed and built on a single heterogeneous graph, which connects all of the documents and words, including the training and testing data. since testing documents must be used in training, transductive gnns cannot be applied to new unseen documents. thus, inductive learning gnns @xcite @xcite have been proposed by representing each document in its own * the corresponding author. graph structure of local word interactions, with pretrained word embedding initialized on each word node. however, the global word embeddings are irrelevant to target documents, and graph structures might not capture the context well since text is usually produced in sequential order. modern transformer-based @xcite ) pretrained models, such as bert @xcite , have shown their effectiveness in capturing context by sequentially modeling documents. in this study, we have also found that bert-style models alone can outperform existing inductive gnns. an intuitive method for enhancing inductive gnns incorporates bert contextual embeddings with gnns for final text classification. for the most relevant research studies, @xcite adopted a gnn on a global vocabulary graph to enrich the token embeddings in bert, and @xcite focused on sentences comparison tasks by feeding bert contextual embeddings into a dependency graph. however, for text classification with inductive gnns, there is a lack of reports on bert and bert-based gnns. only @xcite have adopted bert's document embeddings for transductive gnns. motivated by the recent success of inductive gnns and the strengths of pretrained bert models, in this work, we further consider the fact that these two types of models have different objectives. the former focuses on learning local syntactic word interactions, and the latter captures the context-aware semantics of words. to collaboratively join gnn and bert models, we propose a unified model for learning contextual inductive document representation via graph neural networks, coined contexting, where each model has its own classifier for its own objective. a sub-word graph is adopted in contexting to focus more on finegrained syntactic word usages, such as pre-/post-fix characters, which avoid over-focusing on contentspecific word usages but maintain the flexibility in bert document embeddings contextual embeddings sub-word graph construction sub-word tokenizer context enrichment ... bert classifier token messages passing and aggregation document features graph classifier graph features ˇ y readout ... accommodating to new words. with this study, we make the following contributions: (1) we are the first to highlight the fact that pure gnns are not superior to bert and to provide detailed comparisons of state-of-the-arts (sotas), as well as the model variants. (2) we present contexting, a simple yet effective unification of bert and gnn that yields results superior to pure inductive gnns and bert on a wide range of datasets. (3) we provide few-shot settings to illustrate the model's robustness to unseen words because of the sub-word adaptation. ## contexting this work proposed an unified model that consists of a bert and a gnn modules as shown in figure 1 . contexting seamlessly enriches documentwise contextual information from a bert-style model to the inductive gnn and makes final predictions based on the decisions of the two modules. ## experiments the proposed model is evaluated by addressing these concerns: (1) is contexting better than pure bert-style models and inductive gnns for text classification? (2) can the contexting achieve satisfactory results with limited training data? ## conclusion in this paper, we have proposed contexting, which successfully learns document embeddings sequentially and contextual word interactions nonsequentially at the same time. various context encoders or gnns are also allowed to build contex-ting on top of this framework. in the future, we aim to involve gnns in a large-scale pretraining process in combination with bert.